Oct 30 21:10:43.916603 ip-10-0-137-220 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:10:44.300680 ip-10-0-137-220 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:10:44.300680 ip-10-0-137-220 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:10:44.300680 ip-10-0-137-220 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:10:44.300680 ip-10-0-137-220 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:10:44.305380 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305272 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.305380 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305375 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.305380 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305380 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.305380 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305385 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305390 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305395 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305399 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305403 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305407 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305411 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305414 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305417 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305421 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305425 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305429 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305434 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305439 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305442 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305446 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305449 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305453 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305457 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.305632 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305461 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305465 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305469 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305473 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305477 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305481 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305484 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305488 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305492 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305495 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305500 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305505 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305509 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305513 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305517 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305522 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305527 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305531 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305535 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305540 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.306373 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305544 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305548 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305552 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305556 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305561 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305565 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305569 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305573 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305577 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305581 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305585 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305589 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305593 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305598 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305602 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305606 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305610 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305614 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305618 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305622 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.307172 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305627 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305631 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305635 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305639 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305644 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305648 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305652 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305661 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305668 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305673 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305677 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305682 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305686 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305690 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305694 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305698 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305703 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305707 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305711 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.308005 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305715 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305719 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305725 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305729 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305733 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305737 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305742 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305746 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305750 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305753 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305757 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305761 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305766 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305770 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305774 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305778 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305782 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305786 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305791 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305795 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.308800 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305800 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305804 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305809 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305816 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.305824 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307049 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307059 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307064 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307068 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307073 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307077 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307081 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307085 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307089 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307093 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307098 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307102 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307107 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307111 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307115 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.309412 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307119 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307123 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307127 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307131 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307136 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307140 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307144 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307148 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307152 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307156 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307160 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307164 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307169 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307173 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307177 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307181 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307185 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307189 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307194 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307198 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.309902 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307201 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307206 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307210 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307214 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307219 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307225 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307232 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307237 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307243 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307248 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307252 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307256 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307260 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307264 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307268 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307273 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307277 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307281 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307285 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.310790 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307297 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307301 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307305 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307309 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307329 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307334 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307339 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307343 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307347 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307351 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307356 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307359 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307364 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307368 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307372 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307376 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307381 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307385 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307389 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.311609 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307393 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307397 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307401 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307406 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307414 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307420 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307424 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307428 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307433 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307437 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307441 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307445 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307450 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307454 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307458 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307462 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307466 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307470 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307480 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307484 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.312344 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307491 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307495 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307499 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307503 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307508 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307512 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307515 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307519 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307524 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307528 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307532 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307536 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.307540 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308425 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308436 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308442 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308447 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308451 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308458 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.313044 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308464 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308469 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308473 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308477 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308482 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308486 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308490 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308494 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308498 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308502 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308506 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308513 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308518 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308522 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308526 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308530 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308535 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308539 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308543 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.313621 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308547 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308551 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308555 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308559 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308563 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308567 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308571 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308575 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308579 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308583 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308591 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308597 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308602 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308606 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308611 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308615 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308619 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308624 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308628 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.314074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308632 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308635 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308640 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308643 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308648 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308654 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308658 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308662 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308667 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308670 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308674 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308678 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308682 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308687 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308690 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308694 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308698 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308702 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308715 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308720 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.314622 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308724 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308728 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308732 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308736 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308741 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308745 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308749 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308753 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308757 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308761 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308766 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308769 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308773 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308777 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308782 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308786 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308790 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308796 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308800 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308804 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.315159 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308808 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308812 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308816 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308820 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308824 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308828 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308833 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308837 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308842 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308845 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308850 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308854 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308858 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308862 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308866 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308870 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308874 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308879 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308883 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308887 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.315662 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308893 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.308897 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.309966 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.309981 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.309992 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310000 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310007 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310012 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310019 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310026 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310031 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310036 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310042 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310047 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310052 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310056 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310061 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310066 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310070 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310075 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310080 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310085 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310090 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310095 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:10:44.316118 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310100 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310107 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310112 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310116 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310122 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310126 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310132 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310136 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310141 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310146 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310153 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310158 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310162 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310167 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310171 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310177 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310183 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310191 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310196 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310201 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310205 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310212 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310216 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310221 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310227 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:10:44.316718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310232 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310238 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310243 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310247 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310252 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310256 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310261 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310266 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310271 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310276 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310281 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310286 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310291 2572 flags.go:64] FLAG: --help="false" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310295 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310301 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310306 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310310 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310330 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310336 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310341 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310345 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310350 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310354 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:10:44.317298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310359 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310366 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310370 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310375 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310380 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310385 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310390 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310394 2572 flags.go:64] FLAG: --lock-file="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310399 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310404 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310408 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310417 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310422 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310427 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310431 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310436 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310441 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310446 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310450 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310458 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310462 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310469 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310474 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310478 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310484 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:10:44.317896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310488 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310493 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310497 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310502 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310513 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310519 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310524 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310528 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310535 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310539 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310544 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310549 2572 flags.go:64] FLAG: --port="10250" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310554 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310559 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ded1d443e8932121" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310564 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310568 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310573 2572 flags.go:64] FLAG: --register-node="true" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310578 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310584 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310588 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310593 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310597 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310603 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310608 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310613 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310617 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:10:44.318492 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310622 2572 flags.go:64] FLAG: --runonce="false" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310626 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310631 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310636 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310641 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310645 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310651 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310656 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310661 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310665 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310669 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310677 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310682 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310686 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310695 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310699 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310707 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310712 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310717 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310723 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310727 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310732 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310736 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310741 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310746 2572 flags.go:64] FLAG: --v="2" Oct 30 21:10:44.319077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310753 2572 flags.go:64] FLAG: --version="false" Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310759 2572 flags.go:64] FLAG: --vmodule="" Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310765 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.310770 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.310979 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.310986 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.310992 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.310997 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311001 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311005 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311009 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311013 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311017 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311021 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311027 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311031 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311036 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311040 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311044 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311049 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.319680 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311053 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311060 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311069 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311073 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311079 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311085 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311089 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311094 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311098 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311102 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311106 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311110 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311114 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311119 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311123 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311127 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311131 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311135 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.320171 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311139 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311143 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311147 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311151 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311155 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311159 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311164 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311168 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311173 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311177 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311181 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311186 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311190 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311194 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311202 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311206 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311210 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311214 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311218 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311222 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.320681 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311227 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311231 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311235 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311239 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311244 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311248 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311252 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311256 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311260 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311264 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311268 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311272 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311277 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311281 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311285 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311295 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311300 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311304 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311308 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311326 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.321140 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311331 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311335 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311340 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311344 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311348 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311352 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311361 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311365 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311369 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311373 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311378 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311382 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311386 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311390 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311395 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311399 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311403 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311407 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311411 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311415 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.321624 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311420 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311426 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311432 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311436 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311441 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311446 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311450 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311455 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311459 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311464 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311468 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.311510 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311565 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311572 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311580 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311585 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.322100 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311589 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311593 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311602 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311610 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311615 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311618 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311622 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311626 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311630 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311634 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311638 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311643 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311647 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311652 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311656 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311660 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311664 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311668 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311672 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.322484 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311676 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311680 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311684 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311689 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311693 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311697 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311701 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311706 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311710 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311713 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311717 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311722 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311726 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311731 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311735 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311741 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311746 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311750 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311754 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311759 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.322919 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311763 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311768 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311772 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311776 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311780 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311785 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311790 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311794 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311798 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311801 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311806 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311810 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311814 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311818 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311822 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311826 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311830 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311834 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311838 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311842 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.323414 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311846 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311851 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311855 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311859 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311864 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311868 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311872 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311878 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311884 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311888 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311892 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311897 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311901 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311905 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311910 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311914 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311919 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311923 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311928 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.323868 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311935 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311940 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311944 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311949 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311953 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311957 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311961 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311965 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311969 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311973 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311977 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311981 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311986 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311990 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311994 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.311998 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312002 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312005 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312010 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.324307 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312015 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312021 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312027 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312031 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.312035 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.318948 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.318964 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.318980 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.318984 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319036 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319040 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319044 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319047 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319049 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319052 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319055 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319057 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319060 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319062 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319065 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.324746 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319067 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319070 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319072 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319075 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319079 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319083 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319086 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319089 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319092 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319095 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319098 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319100 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319103 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319105 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319108 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319111 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319113 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319116 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319121 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.325201 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319125 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319129 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319131 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319134 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319137 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319140 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319142 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319145 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319147 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319150 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319152 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319155 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319157 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319160 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319163 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319165 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319167 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319170 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319173 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.325693 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319176 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319178 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319181 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319183 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319186 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319189 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319191 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319194 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319196 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319199 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319202 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319204 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319207 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319210 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319213 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319215 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319218 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319220 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319223 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.326121 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319225 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319228 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319230 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319233 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319235 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319238 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319240 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319243 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319245 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319248 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319250 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319252 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319255 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319258 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319260 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319263 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319265 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319269 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319272 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319274 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.326691 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319277 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319279 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319281 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319284 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319286 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319295 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319299 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319303 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319306 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319308 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319311 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319328 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319331 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319334 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319336 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319339 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319341 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.327173 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319344 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.319371 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319508 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319512 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319515 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319518 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319520 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319523 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319526 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319530 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319533 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319535 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319539 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319542 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319544 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.327589 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319546 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319549 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319552 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319554 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319557 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319559 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319562 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319564 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319567 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319570 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319572 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319574 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319577 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319579 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319582 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319584 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319586 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319589 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319591 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319593 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.327929 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319596 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319598 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319601 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319603 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319605 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319607 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319610 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319612 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319615 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319618 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319621 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319623 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319626 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319628 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319631 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319634 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319637 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319640 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319643 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.328398 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319645 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319648 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319650 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319653 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319655 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319658 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319660 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319663 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319665 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319667 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319670 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319672 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319674 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319677 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319679 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319682 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319684 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319687 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319689 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.328818 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319692 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319695 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319697 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319700 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319703 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319706 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319709 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319711 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319714 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319716 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319718 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319721 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319724 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319726 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319729 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319731 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319734 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319736 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319739 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319741 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319744 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.329243 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319746 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319748 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319751 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319753 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319756 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319758 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319761 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319763 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319766 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319768 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319770 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319773 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319775 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319777 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.319800 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:10:44.329796 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319828 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319831 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319834 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319837 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319840 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319842 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319845 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319847 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319850 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319852 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319854 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319857 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319859 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319862 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319865 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319867 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319870 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319872 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319874 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319877 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:10:44.330151 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319879 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319882 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319884 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319886 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319889 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319891 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319894 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319896 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319899 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319901 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319904 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319906 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319909 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319911 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319914 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319916 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319919 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319921 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319924 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:10:44.330678 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319926 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319929 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319931 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319934 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319936 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319938 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319940 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319944 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319949 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319952 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319955 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319958 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319961 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319963 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319966 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319969 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319971 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319973 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319976 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:10:44.331098 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319978 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319981 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319983 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319985 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319988 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319990 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319995 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.319998 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320000 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320003 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320005 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320008 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320010 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320012 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320015 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320017 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320020 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320022 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320024 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320027 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:10:44.331548 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320029 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320032 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320035 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320037 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320040 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320043 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320045 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320047 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320050 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320052 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320056 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320059 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320062 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320064 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320067 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320069 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320071 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320075 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320079 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:10:44.331989 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320082 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320085 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320087 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320090 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320092 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320095 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320097 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320100 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:44.320102 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.320150 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.322224 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.323224 2572 server.go:1014] "Starting client certificate rotation" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.323331 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.323365 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.325370 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:10:44.332434 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.330343 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:10:44.340183 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.340165 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:10:44.343767 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.343753 2572 log.go:25] "Validated CRI v1 image API" Oct 30 21:10:44.345419 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.345395 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:10:44.349756 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.349738 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 83cc390b-e4e1-4423-9483-88e1eba3af4c:/dev/nvme0n1p4 a848376f-3c81-4d8a-a5da-3b3c510c9ed4:/dev/nvme0n1p3] Oct 30 21:10:44.349813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.349756 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:10:44.354691 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.354550 2572 manager.go:217] Machine: {Timestamp:2026-10-30 21:10:44.353149342 +0000 UTC m=+0.329727022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3190475 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b5677e61b0465162ea12b10f3208 SystemUUID:ec24b567-7e61-b046-5162-ea12b10f3208 BootID:1257dd80-9578-4519-bbca-358b2781137d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:39:11:9d:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:39:11:9d:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:21:59:91:92:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:10:44.354691 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.354685 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:10:44.354822 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.354786 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:10:44.357755 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357730 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:10:44.357878 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357757 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-220.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:10:44.357918 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357881 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:10:44.357918 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357888 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:10:44.357965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357919 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:10:44.357965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.357936 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:10:44.359112 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.359102 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:10:44.362275 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.362264 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:10:44.362400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.362390 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:10:44.363120 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363112 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:10:44.363151 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363124 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:10:44.363151 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363134 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:10:44.363151 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363142 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:10:44.363151 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363151 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:10:44.363505 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.363494 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:10:44.365792 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.365761 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:10:44.373295 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.373279 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:10:44.374219 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374205 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:10:44.374286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374245 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:10:44.374445 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374434 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:10:44.374511 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374453 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:10:44.374511 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374462 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:10:44.374511 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374471 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:10:44.374511 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374479 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:10:44.374511 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.374488 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:10:44.375177 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375167 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:10:44.375232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375179 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:10:44.375232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375190 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:10:44.375232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375199 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:10:44.375232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375211 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:10:44.375232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.375224 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:10:44.376784 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.376772 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:10:44.376848 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.376787 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:10:44.380378 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.380365 2572 server.go:1263] "Started kubelet" Oct 30 21:10:44.380470 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.380440 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:10:44.380532 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.380446 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:10:44.380579 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.380555 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:10:44.381062 ip-10-0-137-220 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:10:44.381634 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.381622 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:10:44.382051 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.382036 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:10:44.388946 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.388925 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:10:44.389123 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.389107 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:10:44.389750 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.389732 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:10:44.389860 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.389823 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:44.390465 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390444 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:10:44.390567 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390444 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:10:44.390567 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390495 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:10:44.390650 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390612 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:10:44.390650 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390621 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:10:44.390807 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390785 2572 factory.go:55] Registering systemd factory Oct 30 21:10:44.390807 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.390806 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:10:44.391024 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.391008 2572 factory.go:153] Registering CRI-O factory Oct 30 21:10:44.391024 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.391022 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 21:10:44.391100 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.391059 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:10:44.391100 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.391073 2572 factory.go:103] Registering Raw factory Oct 30 21:10:44.391100 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.391082 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 21:10:44.392097 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.392081 2572 manager.go:319] Starting recovery of all containers Oct 30 21:10:44.404327 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.404274 2572 manager.go:324] Recovery completed Oct 30 21:10:44.409008 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.408990 2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 21:10:44.411791 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.411780 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.413603 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.413587 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.413676 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.413614 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.413676 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.413623 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.414150 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.414133 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:10:44.414150 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.414148 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:10:44.414250 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.414163 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:10:44.414250 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.414139 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.415174 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.415163 2572 policy_none.go:50] "Start" Oct 30 21:10:44.415210 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.415176 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:10:44.415210 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.415186 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:10:44.416096 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.416086 2572 policy_none.go:44] "Start" Oct 30 21:10:44.420891 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.420873 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.420973 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.420901 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.420973 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.420915 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.450399 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453030 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.453057 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453066 2572 server.go:87] "Starting device plugin registration server" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453284 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453296 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453412 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453521 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.453535 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.454771 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:10:44.457965 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.454814 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:44.460584 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.460566 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:10:44.461606 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.461591 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:10:44.461663 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.461616 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:10:44.461663 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.461646 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:10:44.461781 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.461683 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:10:44.519989 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.519963 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-220.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:10:44.520991 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.519952 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-220.us-west-1.compute.internal.18e36aed6060d2b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-220.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-220.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:10:44.38034092 +0000 UTC m=+0.356918600,LastTimestamp:2026-10-30 21:10:44.38034092 +0000 UTC m=+0.356918600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-220.us-west-1.compute.internal,}" Oct 30 21:10:44.554183 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.554143 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.554876 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.554861 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.554962 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.554892 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.554962 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.554907 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.554962 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.554938 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.562530 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.562508 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal"] Oct 30 21:10:44.562614 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.562561 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.563195 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.563183 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.563259 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.563209 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.563259 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.563224 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.564402 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.564388 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.564530 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.564516 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.564571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.564543 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.565008 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.564993 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.565066 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.564993 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.565066 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.565024 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.565066 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.565034 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.565066 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.565050 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.565234 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.565069 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.566211 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.566196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.566285 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.566226 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.566880 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.566864 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.566949 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.566892 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.566949 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.566904 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.568072 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.568057 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-220.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:10:44.591929 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.591912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.592021 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.591936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.592021 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.591954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.592826 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.592810 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.597708 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.597692 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.601049 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.601034 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:10:44.601120 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.601068 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9nv2t" Oct 30 21:10:44.624593 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.624526 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-220.us-west-1.compute.internal.18e36aed625c5a49 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-220.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-220.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-220.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:10:44.413602377 +0000 UTC m=+0.390180058,LastTimestamp:2026-10-30 21:10:44.413602377 +0000 UTC m=+0.390180058,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-220.us-west-1.compute.internal,}" Oct 30 21:10:44.625061 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.625043 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-220.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:10:44.625106 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.625041 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:10:44.625106 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.625041 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:10:44.674014 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.673994 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 21:10:44.676492 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.676478 2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.692985 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.692970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.693037 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.692996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.693037 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.693015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.693102 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.693049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.693102 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.693053 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.693102 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.693049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.824988 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.824813 2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.877499 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.877483 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:44.878435 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.878419 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:44.878498 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.878456 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:44.878498 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.878472 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:44.878572 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.878499 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.896363 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.896342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.902552 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.902537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.935227 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:44.935210 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:44.935227 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:44.935228 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-220.us-west-1.compute.internal\": node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.042505 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.042485 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.142980 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.142926 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.243894 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.243874 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.323285 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.323267 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:10:45.323660 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.323397 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a37cf823f4ff54d85b23692eedd79e51-8af2da30c05d0787.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-220.us-west-1.compute.internal.18e36aed625c9a45\": read tcp 10.0.137.220:54686->3.139.119.67:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-220.us-west-1.compute.internal.18e36aed625c9a45 default 4784 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-220.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-220.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-220.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:10:44 +0000 UTC,LastTimestamp:2026-10-30 21:10:44.563215189 +0000 UTC m=+0.539792875,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-220.us-west-1.compute.internal,}" Oct 30 21:10:45.344580 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.344553 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.445081 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.445040 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.528432 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.528409 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:10:45.529363 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.529348 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:10:45.529444 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.529378 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:10:45.529444 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.529388 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:10:45.546364 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.546335 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.606127 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.606097 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:05:44 +0000 UTC" deadline="2028-04-14 08:11:20.119050743 +0000 UTC" Oct 30 21:10:45.606127 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.606121 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12755h0m34.512934123s" Oct 30 21:10:45.647065 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.647041 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.701580 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.701556 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:10:45.748576 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.748546 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.849495 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.849468 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.885345 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.885322 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:10:45.949547 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:45.949519 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:10:45.958446 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:45.958380 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48089fcbd21ef30454f56e9b574f5b67.slice/crio-8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b WatchSource:0}: Error finding container 8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b: Status 404 returned error can't find the container with id 8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b Oct 30 21:10:45.958620 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:45.958603 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8005a88bfac70fcc69b4dacc648a18.slice/crio-9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031 WatchSource:0}: Error finding container 9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031: Status 404 returned error can't find the container with id 9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031 Oct 30 21:10:45.961844 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.961825 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:10:45.965181 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.965161 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:10:45.990133 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:45.990103 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:46.056648 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.056631 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:10:46.058592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.058580 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:10:46.121436 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.121416 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:10:46.206290 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.206268 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:10:46.364930 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.364886 2572 apiserver.go:52] "Watching apiserver" Oct 30 21:10:46.392087 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.392066 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:10:46.465817 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.465775 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerStarted","Data":"9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031"} Oct 30 21:10:46.466828 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.466810 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"48089fcbd21ef30454f56e9b574f5b67","Type":"ContainerStarted","Data":"8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b"} Oct 30 21:10:46.477120 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.477101 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:10:46.477406 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.477386 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bq4pk","openshift-multus/multus-jt62z","openshift-multus/network-metrics-daemon-xrkn6","kube-system/global-pull-secret-syncer-mp2j2","openshift-cluster-node-tuning-operator/tuned-q4s6r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal","kube-system/konnectivity-agent-z2rbr","kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt","openshift-multus/multus-additional-cni-plugins-flz96","openshift-network-diagnostics/network-check-target-k67dh","openshift-dns/node-resolver-hqlzb","openshift-network-operator/iptables-alerter-w7cdp","openshift-ovn-kubernetes/ovnkube-node-6d887"] Oct 30 21:10:46.479534 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.479519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.480782 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.480754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.481822 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.481801 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.481913 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.481891 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:46.482762 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.482742 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.482851 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.482807 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:46.483791 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.483773 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.484982 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.484967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.486286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.486272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.487523 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.487508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.488709 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.488695 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.490199 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.490183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.491711 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.491691 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:46.491787 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.491765 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:46.493012 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.492995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.502123 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.502194 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.502194 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.502194 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502176 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.502353 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.502353 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.502353 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.502474 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.502474 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7wt\" (UniqueName: \"kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.502474 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502439 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.502595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhwx\" (UniqueName: \"kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502631 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.502786 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.502989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.503019 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503186 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgvp\" (UniqueName: \"kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.503335 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2tqj\" (UniqueName: \"kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503555 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.503681 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503717 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q59k\" (UniqueName: \"kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvtv\" (UniqueName: \"kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.503997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504068 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504065 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkf9l\" (UniqueName: \"kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504257 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.504405 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ns9z\" (UniqueName: \"kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504532 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.504653 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.504558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.536958 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.536917 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:10:46.537061 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537012 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9kft\"" Oct 30 21:10:46.537164 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537144 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:10:46.537778 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537762 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.537778 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537775 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb6bb\"" Oct 30 21:10:46.537879 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537784 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:10:46.537934 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.537915 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.540039 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.540022 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.540371 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.540354 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.540561 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.540457 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xkwpv\"" Oct 30 21:10:46.541532 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.541516 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:10:46.542125 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.542112 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:10:46.542644 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.542625 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4qh6\"" Oct 30 21:10:46.542718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.542650 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.542718 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.542653 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.543215 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543199 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:10:46.543271 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543220 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.543271 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543242 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:10:46.543353 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543314 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:10:46.543460 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543446 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87wxx\"" Oct 30 21:10:46.543686 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.543675 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.544244 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.544232 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.588545 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.588526 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:10:46.588836 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.588816 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwgx7\"" Oct 30 21:10:46.588966 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.588846 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.591072 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591058 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:10:46.591532 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591489 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:10:46.591721 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591666 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rvh9q\"" Oct 30 21:10:46.591792 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591737 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:10:46.591792 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591741 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.591792 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591788 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.591915 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591886 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-swkpp\"" Oct 30 21:10:46.591915 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591897 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:10:46.591915 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.591897 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:10:46.592236 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.592218 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:10:46.592298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.592250 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:10:46.592298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.592270 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zxj8n" Oct 30 21:10:46.592298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.592279 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wtj2v\"" Oct 30 21:10:46.592298 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.592274 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:10:46.605050 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605115 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.605115 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605115 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkf9l\" (UniqueName: \"kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.605237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.605237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605144 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605164 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.605237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605275 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.605288 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.605361 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.605378 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:10:47.105356163 +0000 UTC m=+3.081933850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605408 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.605429 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:47.105398934 +0000 UTC m=+3.081976621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns9z\" (UniqueName: \"kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.605504 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605547 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605573 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605589 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605631 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605635 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605663 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605699 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605706 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605710 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.605943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605767 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605877 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605884 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605888 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605900 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605910 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605939 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7wt\" (UniqueName: \"kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.605985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.606813 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606100 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhwx\" (UniqueName: \"kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606167 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606137 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606228 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.607617 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606242 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606459 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606555 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606599 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606637 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606680 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606697 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.608515 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgvp\" (UniqueName: \"kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606731 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606758 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606763 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606802 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606831 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606839 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606843 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606882 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606901 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tqj\" (UniqueName: \"kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.609348 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.606973 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607192 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607264 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607195 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607210 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607364 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607378 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607367 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607454 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607478 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.610014 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkfkj\" (UniqueName: \"kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q59k\" (UniqueName: \"kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607575 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607595 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvtv\" (UniqueName: \"kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607651 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607657 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607245 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607733 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607774 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607818 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610536 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607899 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607914 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607974 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607983 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.607996 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.608634 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.608672 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.608828 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.609065 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.609727 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.609999 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.610036 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.610996 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.610648 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.666499 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.666429 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:10:46.666499 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.666471 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:10:46.666499 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.666481 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:46.666698 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:46.666546 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:47.166530554 +0000 UTC m=+3.143108246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:46.668420 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.667989 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns9z\" (UniqueName: \"kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.668809 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.668780 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7wt\" (UniqueName: \"kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:46.668902 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.668874 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhwx\" (UniqueName: \"kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.669057 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.669037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkf9l\" (UniqueName: \"kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.669500 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.669478 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgvp\" (UniqueName: \"kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.669762 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.669736 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tqj\" (UniqueName: \"kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.670307 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.670289 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q59k\" (UniqueName: \"kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.670461 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.670445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvtv\" (UniqueName: \"kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.708883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.708863 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.708960 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.708887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.708960 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.708904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfkj\" (UniqueName: \"kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.709055 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.708995 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.709251 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.709236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.771041 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.771013 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfkj\" (UniqueName: \"kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.791395 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.791364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:46.797084 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.797068 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:10:46.799074 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.799054 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ebb912_daf3_4d1c_b78c_46a14632cca5.slice/crio-37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca WatchSource:0}: Error finding container 37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca: Status 404 returned error can't find the container with id 37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca Oct 30 21:10:46.803978 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.803959 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb110512e_ce89_4a9c_9183_daaf057807d5.slice/crio-9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799 WatchSource:0}: Error finding container 9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799: Status 404 returned error can't find the container with id 9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799 Oct 30 21:10:46.804317 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.804305 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:10:46.813763 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.813739 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c63ac6_3b39_4ee0_a5e1_1985c4d6b146.slice/crio-b9bd2d6bd4132c2e0d944fab85b8db9cb83e76a098a13e92cdb35a738aad9fe1 WatchSource:0}: Error finding container b9bd2d6bd4132c2e0d944fab85b8db9cb83e76a098a13e92cdb35a738aad9fe1: Status 404 returned error can't find the container with id b9bd2d6bd4132c2e0d944fab85b8db9cb83e76a098a13e92cdb35a738aad9fe1 Oct 30 21:10:46.815278 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.815197 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:10:46.821065 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.821047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:10:46.822177 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.822156 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:10:46.822533 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.822510 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2f8f25_7a5a_49fe_818c_67882dd9d616.slice/crio-6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd WatchSource:0}: Error finding container 6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd: Status 404 returned error can't find the container with id 6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd Oct 30 21:10:46.827641 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.827623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt62z" Oct 30 21:10:46.828238 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.828217 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7e01fb_ccea_4561_9de7_7810720748b7.slice/crio-f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0 WatchSource:0}: Error finding container f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0: Status 404 returned error can't find the container with id f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0 Oct 30 21:10:46.831327 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.831309 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75d3c73_4085_468b_ac88_0956a8351130.slice/crio-f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4 WatchSource:0}: Error finding container f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4: Status 404 returned error can't find the container with id f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4 Oct 30 21:10:46.835552 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.835533 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc378851_2cf8_4b37_9ea4_efa2d9027272.slice/crio-39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854 WatchSource:0}: Error finding container 39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854: Status 404 returned error can't find the container with id 39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854 Oct 30 21:10:46.840729 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.840713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:10:46.841827 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:46.841810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:10:46.849908 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.849890 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7b0ca6_fb24_4fd2_b428_fa3bd0d7553b.slice/crio-e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d WatchSource:0}: Error finding container e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d: Status 404 returned error can't find the container with id e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d Oct 30 21:10:46.851235 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:10:46.851216 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bc969b_f10f_4b5b_a52c_788fda6582e4.slice/crio-143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48 WatchSource:0}: Error finding container 143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48: Status 404 returned error can't find the container with id 143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48 Oct 30 21:10:47.111793 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.111756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:47.111955 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.111805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:47.111955 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.111922 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:47.112046 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.111970 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:48.111957254 +0000 UTC m=+4.088534924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:47.112046 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.112014 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:47.112046 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.112037 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:10:48.112029553 +0000 UTC m=+4.088607236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:47.212105 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.212066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:47.212293 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.212243 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:10:47.212293 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.212267 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:10:47.212293 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.212281 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:47.212460 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:47.212354 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:48.212322005 +0000 UTC m=+4.188899690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:47.469622 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.469585 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerStarted","Data":"e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d"} Oct 30 21:10:47.471489 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.471456 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq4pk" event={"ID":"b75d3c73-4085-468b-ac88-0956a8351130","Type":"ContainerStarted","Data":"f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4"} Oct 30 21:10:47.472985 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.472955 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" event={"ID":"8e7e01fb-ccea-4561-9de7-7810720748b7","Type":"ContainerStarted","Data":"f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0"} Oct 30 21:10:47.474454 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.474427 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd"} Oct 30 21:10:47.475670 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.475636 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" event={"ID":"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146","Type":"ContainerStarted","Data":"b9bd2d6bd4132c2e0d944fab85b8db9cb83e76a098a13e92cdb35a738aad9fe1"} Oct 30 21:10:47.476871 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.476846 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7cdp" event={"ID":"b110512e-ce89-4a9c-9183-daaf057807d5","Type":"ContainerStarted","Data":"9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799"} Oct 30 21:10:47.481171 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.481145 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt62z" event={"ID":"dc378851-2cf8-4b37-9ea4-efa2d9027272","Type":"ContainerStarted","Data":"39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854"} Oct 30 21:10:47.482748 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.482718 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z2rbr" event={"ID":"16ebb912-daf3-4d1c-b78c-46a14632cca5","Type":"ContainerStarted","Data":"37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca"} Oct 30 21:10:47.484112 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.484070 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hqlzb" event={"ID":"59bc969b-f10f-4b5b-a52c-788fda6582e4","Type":"ContainerStarted","Data":"143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48"} Oct 30 21:10:47.595074 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.595036 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:05:46 +0000 UTC" deadline="2028-06-01 07:28:25.85705636 +0000 UTC" Oct 30 21:10:47.595074 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:47.595069 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13906h17m38.261991462s" Oct 30 21:10:48.118910 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.118833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:48.118910 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.118879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:48.119123 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.119004 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:48.119123 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.119059 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:50.119042956 +0000 UTC m=+6.095620643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:48.119248 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.119133 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:48.119248 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.119170 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:10:50.119156862 +0000 UTC m=+6.095734550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:48.219584 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.219541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:48.219743 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.219732 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:10:48.219813 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.219755 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:10:48.219813 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.219768 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:48.219917 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.219820 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:50.2198019 +0000 UTC m=+6.196379584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:48.462547 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.462497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:48.463094 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.462871 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:48.463094 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.462967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:48.463094 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.463040 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:48.463512 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.463385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:48.463512 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:48.463478 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:48.596080 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.596015 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:05:46 +0000 UTC" deadline="2028-05-23 15:34:05.814569488 +0000 UTC" Oct 30 21:10:48.596080 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:48.596077 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13698h23m17.218496628s" Oct 30 21:10:49.518476 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:49.518210 2572 generic.go:356] "Generic (PLEG): container finished" podID="6e8005a88bfac70fcc69b4dacc648a18" containerID="b816d015d6194f99d1872e56234c80201363e8d2e1516fd1b5770f0184490ac1" exitCode=0 Oct 30 21:10:49.519184 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:49.519147 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerDied","Data":"b816d015d6194f99d1872e56234c80201363e8d2e1516fd1b5770f0184490ac1"} Oct 30 21:10:49.537264 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:49.536831 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"48089fcbd21ef30454f56e9b574f5b67","Type":"ContainerStarted","Data":"a1547a971b0acd0397d9cbb851dff1d21a244ab9b0123af029aa0e46ecf438ab"} Oct 30 21:10:50.132439 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.132413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:50.132839 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.132448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:50.132839 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.132605 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:50.132839 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.132648 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:54.132636022 +0000 UTC m=+10.109213692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:50.132839 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.132703 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:50.132839 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.132726 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:10:54.132718179 +0000 UTC m=+10.109295849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:50.232954 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.232914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:50.233137 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.233099 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:10:50.233137 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.233120 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:10:50.233137 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.233133 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:50.233295 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.233190 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:54.233169344 +0000 UTC m=+10.209747038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.464508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.464632 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.464706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.464797 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.464837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:50.464977 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:50.464900 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:50.546246 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.545578 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerStarted","Data":"2f688bdfe57323f50d9f3ac372065238fce106463dd482f3b76e7d94a3ae9e6b"} Oct 30 21:10:50.551052 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.550145 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="7ed48fbb06e449dfcaed930871763b3c2d3d01bf2ed432e38146635d1cac024f" exitCode=0 Oct 30 21:10:50.551052 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.550982 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"7ed48fbb06e449dfcaed930871763b3c2d3d01bf2ed432e38146635d1cac024f"} Oct 30 21:10:50.662706 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.661533 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" podStartSLOduration=4.661516463 podStartE2EDuration="4.661516463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:49.774765783 +0000 UTC m=+5.751343479" watchObservedRunningTime="2026-10-30 21:10:50.661516463 +0000 UTC m=+6.638094156" Oct 30 21:10:50.788542 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:50.787639 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" podStartSLOduration=4.787621335 podStartE2EDuration="4.787621335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:50.662523717 +0000 UTC m=+6.639101409" watchObservedRunningTime="2026-10-30 21:10:50.787621335 +0000 UTC m=+6.764199017" Oct 30 21:10:51.555688 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:51.555658 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z2rbr" event={"ID":"16ebb912-daf3-4d1c-b78c-46a14632cca5","Type":"ContainerStarted","Data":"70a308022f41a1fb1d3564ec513802ec759a9ce618136e8040d9a81cd38bcff1"} Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:52.464350 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:52.464452 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:52.464532 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:52.464596 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:52.464651 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:52.465655 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:52.464731 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:53.564668 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.564630 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq4pk" event={"ID":"b75d3c73-4085-468b-ac88-0956a8351130","Type":"ContainerStarted","Data":"e374ea3a98329607d0f7149d6841ad22dc472244eebaa51ec4838a19cda8ad41"} Oct 30 21:10:53.567437 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.567344 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" event={"ID":"8e7e01fb-ccea-4561-9de7-7810720748b7","Type":"ContainerStarted","Data":"f6d08e55091bbd1c6d2d0222f0cc9a3e8c003b702d5d809e85641c826bf001d5"} Oct 30 21:10:53.647972 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.647326 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:53.648652 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.648628 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:10:53.680284 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.680223 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z2rbr" podStartSLOduration=4.118913977 podStartE2EDuration="8.680202163s" totalImagesPullingTime="4.561288186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:51.670860818 +0000 UTC m=+7.647438508" watchObservedRunningTime="2026-10-30 21:10:53.680202163 +0000 UTC m=+9.656779881" Oct 30 21:10:53.797413 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:53.797350 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bq4pk" podStartSLOduration=3.176486979 podStartE2EDuration="8.797329859s" totalImagesPullingTime="5.62084288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:53.681405351 +0000 UTC m=+9.657983040" watchObservedRunningTime="2026-10-30 21:10:53.797329859 +0000 UTC m=+9.773907545" Oct 30 21:10:54.166626 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.166590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:54.166789 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.166637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:54.166825 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.166789 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:54.166861 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.166849 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:02.166831972 +0000 UTC m=+18.143409657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:10:54.167264 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.167242 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:54.167371 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.167295 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:02.167279856 +0000 UTC m=+18.143857538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:10:54.267000 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.266959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:54.267201 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.267182 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:10:54.267285 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.267209 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:10:54.267285 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.267223 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:54.267285 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.267279 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:02.267263331 +0000 UTC m=+18.243841015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:10:54.486641 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.485308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:54.486641 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.486623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:54.488024 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.487947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:54.488264 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:54.488185 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:54.489065 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.489001 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:54.489151 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:54.489123 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:55.108868 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.108835 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:10:55.460652 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.460545 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:10:55.108860678Z","UUID":"133ab96f-f5a5-4dd6-8cdd-078814052178","Handler":null,"Name":"","Endpoint":""} Oct 30 21:10:55.462214 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.462194 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:10:55.462305 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.462220 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:10:55.574192 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.574154 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hqlzb" event={"ID":"59bc969b-f10f-4b5b-a52c-788fda6582e4","Type":"ContainerStarted","Data":"9714785e6e2973535c81c507072e3c6ea7b6498d0658c77101fd54a723c56bd3"} Oct 30 21:10:55.577467 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.577437 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerStarted","Data":"63cec7598f201a3b63c7d1318fc5c4f4d289361488ed866ff970a43a0085485a"} Oct 30 21:10:55.579463 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.579436 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" event={"ID":"8e7e01fb-ccea-4561-9de7-7810720748b7","Type":"ContainerStarted","Data":"ea3027dff068ff4e734fd026033c04b235e589f7ae15a8c143fcc03d2c83ed9c"} Oct 30 21:10:55.581171 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.581143 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt62z" event={"ID":"dc378851-2cf8-4b37-9ea4-efa2d9027272","Type":"ContainerStarted","Data":"9cbbe1b422100f31e2e760bf20ab6532b0691d9b738f553a923b0f825ba2b034"} Oct 30 21:10:55.691585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.691336 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hqlzb" podStartSLOduration=2.280656107 podStartE2EDuration="10.691303056s" totalImagesPullingTime="8.410646949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:55.690836826 +0000 UTC m=+11.667414517" watchObservedRunningTime="2026-10-30 21:10:55.691303056 +0000 UTC m=+11.667880746" Oct 30 21:10:55.926835 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:55.926728 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jt62z" podStartSLOduration=2.837169669 podStartE2EDuration="10.926716205s" totalImagesPullingTime="8.089546536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:55.926406115 +0000 UTC m=+11.902983826" watchObservedRunningTime="2026-10-30 21:10:55.926716205 +0000 UTC m=+11.903293896" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.463447 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.463467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:56.463575 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.463605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:56.463673 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:56.464138 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:56.463753 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:56.586949 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.586111 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="63cec7598f201a3b63c7d1318fc5c4f4d289361488ed866ff970a43a0085485a" exitCode=0 Oct 30 21:10:56.586949 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.586193 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"63cec7598f201a3b63c7d1318fc5c4f4d289361488ed866ff970a43a0085485a"} Oct 30 21:10:56.590937 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.590909 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" event={"ID":"8e7e01fb-ccea-4561-9de7-7810720748b7","Type":"ContainerStarted","Data":"444424b881f183a4f60bdffe009150d2077cab670539feb23fc1504ccb676dc5"} Oct 30 21:10:56.864101 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:56.864047 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" podStartSLOduration=2.652414653 podStartE2EDuration="11.86403236s" totalImagesPullingTime="9.211617707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:56.863770585 +0000 UTC m=+12.840348277" watchObservedRunningTime="2026-10-30 21:10:56.86403236 +0000 UTC m=+12.840610116" Oct 30 21:10:57.594274 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:57.594234 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" event={"ID":"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146","Type":"ContainerStarted","Data":"b5c839c7d2da3b82723d81bc7e7f1028a0ba1dd40f41ef5ca896361fae26b93c"} Oct 30 21:10:57.596645 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:57.596622 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w7cdp" event={"ID":"b110512e-ce89-4a9c-9183-daaf057807d5","Type":"ContainerStarted","Data":"da60be53e7e5ea4843b179da6af92df19a9dbf74dce1a378ca02b1e2c62e19be"} Oct 30 21:10:57.717839 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:57.717767 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" podStartSLOduration=2.0841744269999998 podStartE2EDuration="12.717732098s" totalImagesPullingTime="10.633557671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:57.717232933 +0000 UTC m=+13.693810623" watchObservedRunningTime="2026-10-30 21:10:57.717732098 +0000 UTC m=+13.694309791" Oct 30 21:10:57.835192 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:57.835139 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w7cdp" podStartSLOduration=4.27616993 podStartE2EDuration="12.835118061s" totalImagesPullingTime="8.558948131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:57.835080852 +0000 UTC m=+13.811658545" watchObservedRunningTime="2026-10-30 21:10:57.835118061 +0000 UTC m=+13.811695752" Oct 30 21:10:58.461992 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:58.461957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:10:58.461992 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:58.462002 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:10:58.462233 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:58.462107 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:10:58.462563 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:58.462536 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:10:58.462657 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:58.462609 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:10:58.462720 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:10:58.462694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:10:58.600127 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:58.600090 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="e98fa7e83fb0b791631aff4d413a05600f8d7d06238e0475c56197e28b5287e3" exitCode=0 Oct 30 21:10:58.600705 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:10:58.600173 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"e98fa7e83fb0b791631aff4d413a05600f8d7d06238e0475c56197e28b5287e3"} Oct 30 21:11:00.462825 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:00.462556 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:00.463279 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:00.462628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:00.463279 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:00.462929 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:00.463279 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:00.463035 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:00.463279 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:00.462668 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:00.463279 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:00.463159 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:00.608162 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:00.608125 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="25351818e5fb7ad055e9f1c78bef1a8fd1712c9500897a62a7b706d9c2a2aa74" exitCode=0 Oct 30 21:11:00.608340 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:00.608183 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"25351818e5fb7ad055e9f1c78bef1a8fd1712c9500897a62a7b706d9c2a2aa74"} Oct 30 21:11:02.241352 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.241301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:02.241352 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.241356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:02.241846 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.241537 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:11:02.241846 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.241601 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret podName:503a46bf-3696-4a30-953b-19ed465b5919 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.241581331 +0000 UTC m=+34.218159016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret") pod "global-pull-secret-syncer-mp2j2" (UID: "503a46bf-3696-4a30-953b-19ed465b5919") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:11:02.241846 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.241671 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:11:02.241846 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.241703 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.24169332 +0000 UTC m=+34.218271001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:11:02.342578 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.342497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:02.342736 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.342643 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:11:02.342736 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.342659 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:11:02.342736 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.342670 2572 projected.go:196] Error preparing data for projected volume kube-api-access-sc6w7 for pod openshift-network-diagnostics/network-check-target-k67dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:11:02.342736 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.342724 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7 podName:50585253-3ba0-43a4-a7ca-c38bec0bb1c6 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.342708048 +0000 UTC m=+34.319285721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sc6w7" (UniqueName: "kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7") pod "network-check-target-k67dh" (UID: "50585253-3ba0-43a4-a7ca-c38bec0bb1c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:11:02.462946 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.462913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:02.462946 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.462916 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:02.463167 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.463072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:02.463167 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.463113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:02.463254 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.463225 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:02.463369 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:02.463350 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:02.614773 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.614745 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"82b1a26fcb900cbe156ebddf6e327f6c2449af3f05334a538206df81cf027d80"} Oct 30 21:11:02.614893 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.614827 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"35eaebe9e0ce1b8fad426749c59521d9cd1cc2a6ab459f5d76dce059a5287421"} Oct 30 21:11:02.614893 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.614873 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"a15193f6311bcb754e0622d6c5275ec1a6fbc02ce1616551c5eab1c9df75959a"} Oct 30 21:11:02.615006 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.614921 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"2f872e5c726a39e058b6988630248937ad2598266d6c121f791e62734934501a"} Oct 30 21:11:02.615006 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.614965 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"f07a8c60023569ad9a0cb6c07768cc23be2835523a2dda971ac2ea56197e17d2"} Oct 30 21:11:02.615097 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:02.615032 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"70d6597075a99e967a7a57949c9118f37c3836bde80c638d848f220217902f15"} Oct 30 21:11:03.618924 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:03.618890 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="c8c928034bc6633cd00ff60b282c00a1473145f9459b050b785d7aff33ebd272" exitCode=0 Oct 30 21:11:03.619452 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:03.618949 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"c8c928034bc6633cd00ff60b282c00a1473145f9459b050b785d7aff33ebd272"} Oct 30 21:11:04.462838 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.462806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:04.463024 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.462898 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:04.463024 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:04.462916 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:04.463139 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:04.463049 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:04.463139 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.463086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:04.463216 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:04.463159 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:04.624265 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.624226 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"da55d7f130ce42fae60f2073ea6fbc53622a6c356146b4b30b46e6e7773779e6"} Oct 30 21:11:04.627090 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.627065 2572 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="d48d6a9e540453e3c155d5ffcc663a2e4b9e4b43542e44257d8bab1861ae97ed" exitCode=0 Oct 30 21:11:04.627221 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:04.627103 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerDied","Data":"d48d6a9e540453e3c155d5ffcc663a2e4b9e4b43542e44257d8bab1861ae97ed"} Oct 30 21:11:05.632006 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:05.631971 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flz96" event={"ID":"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b","Type":"ContainerStarted","Data":"1eee2515f54c46d7be9e119de33070cdb94f85a89e199c73832c6a43c29ec816"} Oct 30 21:11:05.855117 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:05.855064 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-flz96" podStartSLOduration=8.152541309 podStartE2EDuration="20.855049701s" totalImagesPullingTime="12.702508392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:05.85469005 +0000 UTC m=+21.831267741" watchObservedRunningTime="2026-10-30 21:11:05.855049701 +0000 UTC m=+21.831627390" Oct 30 21:11:06.462531 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.462325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:06.462717 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.462398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:06.462717 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:06.462630 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:06.462717 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.462441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:06.462717 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:06.462679 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:06.462956 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:06.462745 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:06.637744 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.637641 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" event={"ID":"0b2f8f25-7a5a-49fe-818c-67882dd9d616","Type":"ContainerStarted","Data":"f6d10210249fb0cce609f2f41fe2e90bf52ebd08d686cf916d460e6193e43228"} Oct 30 21:11:06.638224 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.637925 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:06.638224 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.638215 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:06.653622 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.653594 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:06.970232 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:06.970182 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" podStartSLOduration=6.979330879 podStartE2EDuration="21.970163964s" totalImagesPullingTime="14.990833085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:06.843666499 +0000 UTC m=+22.820244187" watchObservedRunningTime="2026-10-30 21:11:06.970163964 +0000 UTC m=+22.946741655" Oct 30 21:11:07.640269 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:07.640243 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:07.654558 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:07.654535 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:08.461835 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.461806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:08.462148 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.461843 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:08.462148 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.461891 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:08.462148 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.461973 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:08.462148 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.462038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:08.462148 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.462114 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:08.921804 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.921721 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mp2j2"] Oct 30 21:11:08.922185 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.921838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:08.922185 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.921931 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:08.922592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.922570 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k67dh"] Oct 30 21:11:08.922699 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.922645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:08.922762 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.922710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:08.924769 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.924750 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrkn6"] Oct 30 21:11:08.924832 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:08.924824 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:08.924906 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:08.924893 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:09.000851 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:09.000821 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:09.000997 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:09.000964 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:11:09.001463 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:09.001445 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:10.462920 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:10.462889 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:10.462920 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:10.462906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:10.463385 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:10.463012 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:10.463385 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:10.463033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:10.463385 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:10.463109 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:10.463385 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:10.463178 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:12.462237 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:12.462201 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:12.462770 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:12.462245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:12.462770 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:12.462201 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:12.462770 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:12.462311 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:12.462770 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:12.462401 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:12.462770 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:12.462490 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:14.463365 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.463331 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:14.463817 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.463425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:14.463817 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.463468 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:14.463817 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:14.463532 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrkn6" podUID="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" Oct 30 21:11:14.463817 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:14.463546 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67dh" podUID="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" Oct 30 21:11:14.463817 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:14.463622 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mp2j2" podUID="503a46bf-3696-4a30-953b-19ed465b5919" Oct 30 21:11:14.851613 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.851427 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeReady" Oct 30 21:11:14.851775 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.851706 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:11:14.967025 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.966993 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmkfl"] Oct 30 21:11:14.968791 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:14.968771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:15.017583 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.017559 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4vfv6"] Oct 30 21:11:15.019163 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.019148 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm"] Oct 30 21:11:15.019285 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.019270 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:15.020716 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.020699 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rvc5s"] Oct 30 21:11:15.020852 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.020834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.022193 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.022175 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89"] Oct 30 21:11:15.022329 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.022300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.023873 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.023855 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-44tx6"] Oct 30 21:11:15.023967 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.023952 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.026834 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.026812 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs27v"] Oct 30 21:11:15.028415 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.028390 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:15.028943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.028924 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j"] Oct 30 21:11:15.029025 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.029011 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.030797 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.030782 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6dc5bbb4f7-hmkgf"] Oct 30 21:11:15.030916 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.030903 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.030995 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.030976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2n2k\" (UniqueName: \"kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k\") pod \"kas-connection-checker-685895c7c9-xmkfl\" (UID: \"0620bbf3-fe2f-4967-b303-810f585e75e5\") " pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:15.033340 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.032730 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8"] Oct 30 21:11:15.033340 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.032846 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.034443 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.034425 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bt4k"] Oct 30 21:11:15.034552 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.034539 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.036165 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.036149 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw"] Oct 30 21:11:15.036271 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.036256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.037853 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.037836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:15.068138 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.068118 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.068236 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.068118 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.068555 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.068531 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t"] Oct 30 21:11:15.071640 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.071623 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg"] Oct 30 21:11:15.071830 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.071715 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.073978 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.073958 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:11:15.074077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.073958 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tzzvv\"" Oct 30 21:11:15.074077 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.073962 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b9ddc8f6f-shs2w"] Oct 30 21:11:15.074181 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.074125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:15.074397 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.074372 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:11:15.075633 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.075617 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dspqr"] Oct 30 21:11:15.075733 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.075720 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.077296 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.077282 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qmnhw"] Oct 30 21:11:15.077416 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.077401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.079038 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.079022 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4vfv6"] Oct 30 21:11:15.079127 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.079050 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmkfl"] Oct 30 21:11:15.079127 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.079064 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rvc5s"] Oct 30 21:11:15.079228 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.079148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.082186 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.082164 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bt4k"] Oct 30 21:11:15.086003 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.085734 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j"] Oct 30 21:11:15.088551 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.088527 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-44tx6"] Oct 30 21:11:15.089374 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.089356 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw"] Oct 30 21:11:15.092858 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.092839 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dc5bbb4f7-hmkgf"] Oct 30 21:11:15.095796 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.095719 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg"] Oct 30 21:11:15.119218 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.119196 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs27v"] Oct 30 21:11:15.119982 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.119963 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8"] Oct 30 21:11:15.120772 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.120750 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dspqr"] Oct 30 21:11:15.124860 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.124842 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.124860 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.124856 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125047 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.124877 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:11:15.125047 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.124976 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:11:15.125047 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.124978 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125224 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125190 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125445 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125233 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:11:15.125445 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125259 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125445 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125267 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125445 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125412 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:11:15.125628 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125462 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:11:15.125628 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125502 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125628 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125594 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125738 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125656 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:11:15.125738 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125703 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:11:15.125822 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125799 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125854 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125828 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125854 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125837 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125905 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125897 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125905 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125901 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.125963 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125941 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125963 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125942 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.125963 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.125954 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.132592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.132735 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.132735 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.132735 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frqj\" (UniqueName: \"kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj\") pod \"kas-connection-checker-685895c7c9-44tx6\" (UID: \"e9686531-8650-4b17-a9f8-002a96ba1fb4\") " pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:15.132895 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.132895 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132772 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.132895 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2n2k\" (UniqueName: \"kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k\") pod \"kas-connection-checker-685895c7c9-xmkfl\" (UID: \"0620bbf3-fe2f-4967-b303-810f585e75e5\") " pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:15.132895 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132858 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.132895 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.132994 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwzx\" (UniqueName: \"kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx\") pod \"kas-connection-checker-685895c7c9-4vfv6\" (UID: \"efcd9b7e-a9d6-49c1-8792-69fec47961ee\") " pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133070 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133098 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpt6\" (UniqueName: \"kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.133126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133199 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2g7\" (UniqueName: \"kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrkq\" (UniqueName: \"kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2d69\" (UniqueName: \"kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133308 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4pp\" (UniqueName: \"kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133497 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.133571 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133591 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghnh\" (UniqueName: \"kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmm6d\" (UniqueName: \"kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d\") pod \"volume-data-source-validator-58c9f5f46d-6s6mw\" (UID: \"d120a1ef-dae4-4199-8149-0ae9660d8dfe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.133937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.134072 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.134242 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.134174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcp5\" (UniqueName: \"kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.175651 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.175621 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:11:15.177450 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.177410 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:11:15.177589 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.177574 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:11:15.178223 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.178206 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm"] Oct 30 21:11:15.186707 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.186692 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89"] Oct 30 21:11:15.187819 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.187800 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmnhw"] Oct 30 21:11:15.188442 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.188427 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t"] Oct 30 21:11:15.189195 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.189178 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:11:15.189287 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.189196 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:11:15.189287 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.189198 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:11:15.189287 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.189218 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:11:15.190150 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.190135 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b9ddc8f6f-shs2w"] Oct 30 21:11:15.193208 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193187 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193235 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6vs6h\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193245 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rvqfx\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193247 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193272 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-46nl2\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193279 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193291 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:11:15.193342 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193278 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193375 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dssbm\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193375 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qm8dr\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193414 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8ztc5\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193467 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9dh6n\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193517 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:11:15.193592 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.193587 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m4h2f\"" Oct 30 21:11:15.195868 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.195855 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k8b2g\"" Oct 30 21:11:15.196148 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.196135 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:11:15.196202 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.196149 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:11:15.200083 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.200067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2n2k\" (UniqueName: \"kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k\") pod \"kas-connection-checker-685895c7c9-xmkfl\" (UID: \"0620bbf3-fe2f-4967-b303-810f585e75e5\") " pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:15.226532 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.226450 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.226532 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.226497 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:11:15.226700 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.226545 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.226919 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.226886 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:11:15.226919 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.226913 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:11:15.228006 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.227993 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:11:15.234700 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.234750 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.234750 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.234815 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.234938 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.234970 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.235020 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.234985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.235069 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwzx\" (UniqueName: \"kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx\") pod \"kas-connection-checker-685895c7c9-4vfv6\" (UID: \"efcd9b7e-a9d6-49c1-8792-69fec47961ee\") " pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:15.235113 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.235071 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:15.235113 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.235217 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.235132 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.73511119 +0000 UTC m=+31.711688875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:15.235217 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235165 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.235217 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpt6\" (UniqueName: \"kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t56r\" (UniqueName: \"kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235335 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.235400 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.235703 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.235519 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:15.235703 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.235599 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.735582023 +0000 UTC m=+31.712159711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:15.235703 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.235703 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235668 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.235703 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235668 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.235964 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2g7\" (UniqueName: \"kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.235964 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235779 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8g8\" (UniqueName: \"kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8\") pod \"network-check-source-6f78d59f99-rv9lg\" (UID: \"1a68f610-9f32-485e-8814-8e624d1c1538\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:15.235964 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.235964 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrkq\" (UniqueName: \"kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.235964 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.236225 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.235992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.236225 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2d69\" (UniqueName: \"kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.236225 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.236118 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.736101907 +0000 UTC m=+31.712679594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:15.236225 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236161 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.236507 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.236507 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236356 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.236507 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236364 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.236507 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.236939 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.236903 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.237236 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.237388 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.237388 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.237388 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4pp\" (UniqueName: \"kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.237388 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237406 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.237437 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.237513 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.737473197 +0000 UTC m=+31.714050886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:15.237587 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rf6n\" (UniqueName: \"kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghnh\" (UniqueName: \"kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237694 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmm6d\" (UniqueName: \"kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d\") pod \"volume-data-source-validator-58c9f5f46d-6s6mw\" (UID: \"d120a1ef-dae4-4199-8149-0ae9660d8dfe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcp5\" (UniqueName: \"kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.237883 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.237976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frqj\" (UniqueName: \"kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj\") pod \"kas-connection-checker-685895c7c9-44tx6\" (UID: \"e9686531-8650-4b17-a9f8-002a96ba1fb4\") " pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238027 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.238303 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.238833 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238307 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.238833 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238414 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.238833 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.238587 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.239025 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239001 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.239084 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.239138 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239078 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.239192 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.239157 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:15.239192 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.239172 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:15.239289 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.239213 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.739198518 +0000 UTC m=+31.715776189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:15.239422 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239399 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.239777 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.239498 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:15.239777 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.239548 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.739535298 +0000 UTC m=+31.716112979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:15.239777 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.240031 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239784 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.240031 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.239876 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.240132 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.240056 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.240236 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.240215 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.240551 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.240528 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.241087 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.241067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.241454 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.241439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.278542 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.278518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:15.296141 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.296115 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpt6\" (UniqueName: \"kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.296947 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.296836 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwzx\" (UniqueName: \"kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx\") pod \"kas-connection-checker-685895c7c9-4vfv6\" (UID: \"efcd9b7e-a9d6-49c1-8792-69fec47961ee\") " pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:15.297308 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.297277 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2g7\" (UniqueName: \"kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.297419 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.297366 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrkq\" (UniqueName: \"kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.297808 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.297786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.300497 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.300024 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2d69\" (UniqueName: \"kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.300612 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.300510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcp5\" (UniqueName: \"kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.300830 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.300429 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4pp\" (UniqueName: \"kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.300830 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.300769 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.301152 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.301129 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmm6d\" (UniqueName: \"kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d\") pod \"volume-data-source-validator-58c9f5f46d-6s6mw\" (UID: \"d120a1ef-dae4-4199-8149-0ae9660d8dfe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:15.302766 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.302744 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frqj\" (UniqueName: \"kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj\") pod \"kas-connection-checker-685895c7c9-44tx6\" (UID: \"e9686531-8650-4b17-a9f8-002a96ba1fb4\") " pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:15.304552 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.304522 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghnh\" (UniqueName: \"kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.332083 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.332047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:15.339087 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rf6n\" (UniqueName: \"kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.339286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339286 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t56r\" (UniqueName: \"kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8g8\" (UniqueName: \"kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8\") pod \"network-check-source-6f78d59f99-rv9lg\" (UID: \"1a68f610-9f32-485e-8814-8e624d1c1538\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:15.339585 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.340136 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.340136 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.339754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.340638 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.340659 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.340712 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.840693379 +0000 UTC m=+31.817271057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.340970 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.341017 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:15.841002156 +0000 UTC m=+31.817579846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.341278 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.341534 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.341771 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.342284 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.342134 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.343078 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.343046 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.345085 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.345069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:15.345430 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.345410 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.346269 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.346246 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.356804 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.356777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:15.362516 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.362337 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:15.368600 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.368577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:15.401334 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.400614 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8g8\" (UniqueName: \"kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8\") pod \"network-check-source-6f78d59f99-rv9lg\" (UID: \"1a68f610-9f32-485e-8814-8e624d1c1538\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:15.401858 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.401570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:15.405913 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.405821 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.407119 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.407073 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rf6n\" (UniqueName: \"kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.409394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.409298 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t56r\" (UniqueName: \"kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.412693 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.412658 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.418932 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.418667 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:15.440459 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.435698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:15.659737 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.659602 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmkfl"] Oct 30 21:11:15.662068 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.662042 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0620bbf3_fe2f_4967_b303_810f585e75e5.slice/crio-7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb WatchSource:0}: Error finding container 7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb: Status 404 returned error can't find the container with id 7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb Oct 30 21:11:15.725976 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.725951 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm"] Oct 30 21:11:15.731223 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.731193 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53efca0c_3152_434c_998e_d0916b982591.slice/crio-abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676 WatchSource:0}: Error finding container abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676: Status 404 returned error can't find the container with id abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676 Oct 30 21:11:15.742000 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.741977 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-44tx6"] Oct 30 21:11:15.743048 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:15.743126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:15.743126 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.743222 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:15.743222 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743195 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:15.743222 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743209 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:15.743305 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743244 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:15.743305 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743260 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743241584 +0000 UTC m=+32.719819272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:15.743305 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743246 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:15.743305 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743293 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743288204 +0000 UTC m=+32.719865872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:15.743305 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743195 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:15.743468 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:15.743468 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743339 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743325995 +0000 UTC m=+32.719903677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:15.743468 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743359 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743353482 +0000 UTC m=+32.719931154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:15.743468 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743371 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743366038 +0000 UTC m=+32.719943706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:15.743468 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.743416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:15.743606 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743486 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:15.743606 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.743512 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.743506136 +0000 UTC m=+32.720083835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:15.746131 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.746112 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9686531_8650_4b17_a9f8_002a96ba1fb4.slice/crio-89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f WatchSource:0}: Error finding container 89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f: Status 404 returned error can't find the container with id 89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f Oct 30 21:11:15.756963 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.756940 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4vfv6"] Oct 30 21:11:15.759651 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.759622 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcd9b7e_a9d6_49c1_8792_69fec47961ee.slice/crio-98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b WatchSource:0}: Error finding container 98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b: Status 404 returned error can't find the container with id 98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b Oct 30 21:11:15.778399 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.778298 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rvc5s"] Oct 30 21:11:15.781627 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.781599 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b295f49_70ab_4d12_b08c_57f3dfb5324b.slice/crio-210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2 WatchSource:0}: Error finding container 210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2: Status 404 returned error can't find the container with id 210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2 Oct 30 21:11:15.813696 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.813661 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-rs27v"] Oct 30 21:11:15.819137 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.819110 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0ca892_6436_4c1b_994c_bb97ba977307.slice/crio-6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec WatchSource:0}: Error finding container 6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec: Status 404 returned error can't find the container with id 6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec Oct 30 21:11:15.844044 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.844022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:15.844144 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.844135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:15.844212 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.844178 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:15.844258 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.844241 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.844223538 +0000 UTC m=+32.820801207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:15.844258 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.844247 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:15.844258 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.844258 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:15.844377 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:15.844293 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:16.844281051 +0000 UTC m=+32.820858731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:15.858122 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.858093 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw"] Oct 30 21:11:15.858915 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.858865 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg"] Oct 30 21:11:15.861752 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.861598 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd120a1ef_dae4_4199_8149_0ae9660d8dfe.slice/crio-45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880 WatchSource:0}: Error finding container 45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880: Status 404 returned error can't find the container with id 45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880 Oct 30 21:11:15.862523 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.862501 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a68f610_9f32_485e_8814_8e624d1c1538.slice/crio-0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94 WatchSource:0}: Error finding container 0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94: Status 404 returned error can't find the container with id 0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94 Oct 30 21:11:15.877269 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.877246 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qmnhw"] Oct 30 21:11:15.880212 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.880189 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aafa0f5_c7f7_40fc_a6b9_6b4d551bcca2.slice/crio-711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c WatchSource:0}: Error finding container 711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c: Status 404 returned error can't find the container with id 711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c Oct 30 21:11:15.930566 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:15.930543 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j"] Oct 30 21:11:15.933187 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:15.933167 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bb38ba_2a52_40fc_9b88_a21200c1c542.slice/crio-13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6 WatchSource:0}: Error finding container 13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6: Status 404 returned error can't find the container with id 13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6 Oct 30 21:11:16.462677 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.462642 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:16.462862 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.462643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:16.462862 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.462799 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:16.516607 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.516584 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:11:16.516795 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.516778 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8s65\"" Oct 30 21:11:16.516834 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.516781 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z8gbk\"" Oct 30 21:11:16.516867 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.516833 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:11:16.516899 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.516888 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tklwz\"" Oct 30 21:11:16.659582 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.659553 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" event={"ID":"89bb38ba-2a52-40fc-9b88-a21200c1c542","Type":"ContainerStarted","Data":"13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6"} Oct 30 21:11:16.661031 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.661007 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-44tx6" event={"ID":"e9686531-8650-4b17-a9f8-002a96ba1fb4","Type":"ContainerStarted","Data":"cb17be3803f4c9f5a81cf1f46b2239af188bd265103ab7ae259477b80a8e1cfd"} Oct 30 21:11:16.661439 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.661057 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-44tx6" event={"ID":"e9686531-8650-4b17-a9f8-002a96ba1fb4","Type":"ContainerStarted","Data":"89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f"} Oct 30 21:11:16.662595 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.662571 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" event={"ID":"efcd9b7e-a9d6-49c1-8792-69fec47961ee","Type":"ContainerStarted","Data":"66ff33b84f36ec4b90163e9af5e193d60ffd769da7783dce31131f67d6fcd149"} Oct 30 21:11:16.662680 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.662615 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" event={"ID":"efcd9b7e-a9d6-49c1-8792-69fec47961ee","Type":"ContainerStarted","Data":"98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b"} Oct 30 21:11:16.663809 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.663789 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmnhw" event={"ID":"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2","Type":"ContainerStarted","Data":"711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c"} Oct 30 21:11:16.664896 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.664861 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" event={"ID":"1a68f610-9f32-485e-8814-8e624d1c1538","Type":"ContainerStarted","Data":"0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94"} Oct 30 21:11:16.666103 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.666072 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" event={"ID":"d120a1ef-dae4-4199-8149-0ae9660d8dfe","Type":"ContainerStarted","Data":"45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880"} Oct 30 21:11:16.667158 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.667140 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerStarted","Data":"210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2"} Oct 30 21:11:16.668628 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.668440 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" event={"ID":"53efca0c-3152-434c-998e-d0916b982591","Type":"ContainerStarted","Data":"abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676"} Oct 30 21:11:16.669722 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.669683 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs27v" event={"ID":"5f0ca892-6436-4c1b-994c-bb97ba977307","Type":"ContainerStarted","Data":"6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec"} Oct 30 21:11:16.671347 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.671305 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" event={"ID":"0620bbf3-fe2f-4967-b303-810f585e75e5","Type":"ContainerStarted","Data":"81a8587ca7d0f6c1ba5d9f6766d235ee174d5d2dda9daaaa0b011992e774354f"} Oct 30 21:11:16.671429 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.671367 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" event={"ID":"0620bbf3-fe2f-4967-b303-810f585e75e5","Type":"ContainerStarted","Data":"7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb"} Oct 30 21:11:16.752189 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:16.752189 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:16.752387 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752237 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:16.752387 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752299 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:16.752387 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752332 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:16.752387 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752304 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.752286251 +0000 UTC m=+34.728863941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:16.752387 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752369 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.752361254 +0000 UTC m=+34.728938925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:16.752535 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:16.752535 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:16.752535 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752502 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:16.752535 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:16.752646 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.752564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:16.752646 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752590 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:16.752646 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752622 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.752610577 +0000 UTC m=+34.729188259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:16.752731 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752671 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:16.752731 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752704 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.75269457 +0000 UTC m=+34.729272253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:16.752788 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752765 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.75275611 +0000 UTC m=+34.729333779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:16.752788 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.752786 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.752777769 +0000 UTC m=+34.729355439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.853034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:16.853190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.853396 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.853414 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.853501 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.853481151 +0000 UTC m=+34.830058847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.853564 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:16.853613 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:16.853587 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:18.85358081 +0000 UTC m=+34.830158491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:17.162735 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:17.162636 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" podStartSLOduration=426.162617355 podStartE2EDuration="7m6.162617355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:17.16141495 +0000 UTC m=+33.137992643" watchObservedRunningTime="2026-10-30 21:11:17.162617355 +0000 UTC m=+33.139195046" Oct 30 21:11:17.277667 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:17.277594 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" podStartSLOduration=426.277574567 podStartE2EDuration="7m6.277574567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:17.276248332 +0000 UTC m=+33.252826026" watchObservedRunningTime="2026-10-30 21:11:17.277574567 +0000 UTC m=+33.254152260" Oct 30 21:11:18.270158 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.270110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:18.270158 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.270160 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:18.270863 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.270650 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:18.270863 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.270723 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:50.270704186 +0000 UTC m=+66.247281855 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:18.277008 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.276955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:18.371934 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.371890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:18.378037 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.377966 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:18.576570 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.576148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:18.588935 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.588736 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:18.705210 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.705038 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" event={"ID":"d120a1ef-dae4-4199-8149-0ae9660d8dfe","Type":"ContainerStarted","Data":"b2c04615d13d9631fceec2609b72ab4c2a6b6ef3ccb436c6c247199526a60a5d"} Oct 30 21:11:18.781893 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.781838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:18.782054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.781916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:18.782054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.781942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:18.782054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.781984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:18.782054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.782014 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:18.782054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.782049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:18.782221 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782138 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:18.782221 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782183 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782171276 +0000 UTC m=+38.758748957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:18.782444 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782426 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:18.782491 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782481 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782466679 +0000 UTC m=+38.759044364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:18.782544 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782532 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782521348 +0000 UTC m=+38.759099032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:18.782591 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782543 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:18.782591 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782557 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:18.782591 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782589 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782579071 +0000 UTC m=+38.759156756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:18.782687 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782591 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:18.782687 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782622 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782613534 +0000 UTC m=+38.759191217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:18.782687 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782638 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:18.782687 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.782684 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.782660629 +0000 UTC m=+38.759238338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:18.821251 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.820592 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-44tx6" podStartSLOduration=427.820570995 podStartE2EDuration="7m7.820570995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:17.393996978 +0000 UTC m=+33.370574669" watchObservedRunningTime="2026-10-30 21:11:18.820570995 +0000 UTC m=+34.797148686" Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.882993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:18.883153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.883294 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.883310 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.883379 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.883362114 +0000 UTC m=+38.859939797 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.883451 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:18.884176 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:18.883482 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:22.883472419 +0000 UTC m=+38.860050103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:19.010950 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:19.010197 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" podStartSLOduration=391.850333943 podStartE2EDuration="6m34.010174705s" totalImagesPullingTime="2.159840762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:18.821197916 +0000 UTC m=+34.797775607" watchObservedRunningTime="2026-10-30 21:11:19.010174705 +0000 UTC m=+34.986752398" Oct 30 21:11:19.010950 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:19.010873 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mp2j2"] Oct 30 21:11:19.014542 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:19.014488 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503a46bf_3696_4a30_953b_19ed465b5919.slice/crio-e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec WatchSource:0}: Error finding container e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec: Status 404 returned error can't find the container with id e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec Oct 30 21:11:19.062956 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:19.062914 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k67dh"] Oct 30 21:11:19.716010 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:19.714620 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mp2j2" event={"ID":"503a46bf-3696-4a30-953b-19ed465b5919","Type":"ContainerStarted","Data":"e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec"} Oct 30 21:11:19.717375 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:19.717090 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k67dh" event={"ID":"50585253-3ba0-43a4-a7ca-c38bec0bb1c6","Type":"ContainerStarted","Data":"6dbddf6eea81b44225ac548bf6186292f060916754fe7de8ef512076e7867093"} Oct 30 21:11:20.723054 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:20.723019 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/0.log" Oct 30 21:11:20.723533 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:20.723071 2572 generic.go:356] "Generic (PLEG): container finished" podID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerID="21efdc9ea3ed273c7893c0ddc256169d09326bf8cf53e0f210c95aca12284a6e" exitCode=255 Oct 30 21:11:20.723533 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:20.723131 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerDied","Data":"21efdc9ea3ed273c7893c0ddc256169d09326bf8cf53e0f210c95aca12284a6e"} Oct 30 21:11:20.723533 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:20.723421 2572 scope.go:122] "RemoveContainer" containerID="21efdc9ea3ed273c7893c0ddc256169d09326bf8cf53e0f210c95aca12284a6e" Oct 30 21:11:21.731295 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.730553 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qmnhw" event={"ID":"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2","Type":"ContainerStarted","Data":"138944fc5a6a34f478b71e3b9230d7634062a1abe2b7b80539b70cd2831ec691"} Oct 30 21:11:21.739584 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.739497 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:21.740382 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.740186 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/0.log" Oct 30 21:11:21.740382 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.740213 2572 generic.go:356] "Generic (PLEG): container finished" podID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" exitCode=255 Oct 30 21:11:21.740382 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.740263 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerDied","Data":"137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5"} Oct 30 21:11:21.740382 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.740299 2572 scope.go:122] "RemoveContainer" containerID="21efdc9ea3ed273c7893c0ddc256169d09326bf8cf53e0f210c95aca12284a6e" Oct 30 21:11:21.740693 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.740659 2572 scope.go:122] "RemoveContainer" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" Oct 30 21:11:21.740951 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:21.740865 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:21.745708 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.745685 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" event={"ID":"53efca0c-3152-434c-998e-d0916b982591","Type":"ContainerStarted","Data":"33c91b02c42a80bcb81ca94edcb69ecb3de1c8c967f2a54e07375501182dd905"} Oct 30 21:11:21.751601 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.751577 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs27v" event={"ID":"5f0ca892-6436-4c1b-994c-bb97ba977307","Type":"ContainerStarted","Data":"f9d88ef1c437244ed299faf480d67e7ae34960ad13a72d6b7d3646b1712a0a17"} Oct 30 21:11:21.757267 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.757246 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" event={"ID":"89bb38ba-2a52-40fc-9b88-a21200c1c542","Type":"ContainerStarted","Data":"09da65e64f182dd55efcf9a9b734c0abf641dcfa0b361dbc57e7b400c59e2a3a"} Oct 30 21:11:21.851293 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:21.851198 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qmnhw" podStartSLOduration=2.127475504 podStartE2EDuration="7.85117827s" totalImagesPullingTime="5.723702766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:21.850013865 +0000 UTC m=+37.826591576" watchObservedRunningTime="2026-10-30 21:11:21.85117827 +0000 UTC m=+37.827755961" Oct 30 21:11:22.084707 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.084656 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-rs27v" podStartSLOduration=388.99319894 podStartE2EDuration="6m34.084640844s" totalImagesPullingTime="5.091441904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:22.083905192 +0000 UTC m=+38.060482883" watchObservedRunningTime="2026-10-30 21:11:22.084640844 +0000 UTC m=+38.061218532" Oct 30 21:11:22.323657 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.322757 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" podStartSLOduration=386.118642067 podStartE2EDuration="6m31.32273615s" totalImagesPullingTime="5.204094083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:22.204525622 +0000 UTC m=+38.181103313" watchObservedRunningTime="2026-10-30 21:11:22.32273615 +0000 UTC m=+38.299313841" Oct 30 21:11:22.323657 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.323275 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" podStartSLOduration=388.322377327 podStartE2EDuration="6m33.323267885s" totalImagesPullingTime="5.000890558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:22.321404524 +0000 UTC m=+38.297982237" watchObservedRunningTime="2026-10-30 21:11:22.323267885 +0000 UTC m=+38.299845576" Oct 30 21:11:22.450905 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.450869 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-76zdg"] Oct 30 21:11:22.482067 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.482038 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.512954 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.512919 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-76zdg"] Oct 30 21:11:22.568050 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.568014 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:11:22.568219 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.568086 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2dlm\"" Oct 30 21:11:22.568219 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.568122 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:11:22.631058 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.630664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.631058 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.630728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdtc\" (UniqueName: \"kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.631058 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.630770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.631058 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.630810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.631058 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.630885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.731877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.732011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.732064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdtc\" (UniqueName: \"kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.732107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.732148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.732502 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.733239 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.733421 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:23.233400188 +0000 UTC m=+39.209977870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : configmap "kube-rbac-proxy" not found Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.733351 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.733542 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:23.233518519 +0000 UTC m=+39.210096203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:22.744394 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.733803 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.765456 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.764341 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" event={"ID":"1a68f610-9f32-485e-8814-8e624d1c1538","Type":"ContainerStarted","Data":"60a4f74a000e4e553edd8a131edfad9ba44485e0d27f0e371b180d2ce3a30994"} Oct 30 21:11:22.766527 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.766502 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:22.766872 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.766851 2572 scope.go:122] "RemoveContainer" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" Oct 30 21:11:22.767109 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.767086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:22.793541 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.793514 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdtc\" (UniqueName: \"kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:22.833295 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:22.833487 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833340 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:22.833487 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:22.833487 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833448 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833458 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.833437199 +0000 UTC m=+46.810014881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833504 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833525 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833549 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.833534753 +0000 UTC m=+46.810112422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833575 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.833565782 +0000 UTC m=+46.810143454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833595 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833608 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833645 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.833632326 +0000 UTC m=+46.810210010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:22.833652 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:22.833961 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.833695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:22.833961 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833794 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:22.833961 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833836 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:22.833961 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833844 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.83383023 +0000 UTC m=+46.810407913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:22.833961 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.833876 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.833863381 +0000 UTC m=+46.810441065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:22.883716 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.883652 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" podStartSLOduration=391.655957963 podStartE2EDuration="6m37.883632656s" totalImagesPullingTime="6.227674693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:22.882506443 +0000 UTC m=+38.859084135" watchObservedRunningTime="2026-10-30 21:11:22.883632656 +0000 UTC m=+38.860210349" Oct 30 21:11:22.935480 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.935446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:22.935637 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.935547 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:22.935637 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.935560 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:22.935637 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.935608 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.935592726 +0000 UTC m=+46.912170395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:22.935747 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.935665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:22.935789 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.935749 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:22.935789 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:22.935782 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:30.93577313 +0000 UTC m=+46.912350798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:22.941561 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.941518 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4"] Oct 30 21:11:22.957647 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:22.957622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:23.000500 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.000425 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4"] Oct 30 21:11:23.087672 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.087630 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:11:23.087863 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.087713 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:23.087863 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.087741 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mfb44\"" Oct 30 21:11:23.137327 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.137288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbvl\" (UniqueName: \"kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl\") pod \"migrator-66df5d496b-n85d4\" (UID: \"b88c250c-3bfc-4b74-a0b3-b02be06b0f78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:23.238341 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.238177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:23.238341 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.238222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbvl\" (UniqueName: \"kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl\") pod \"migrator-66df5d496b-n85d4\" (UID: \"b88c250c-3bfc-4b74-a0b3-b02be06b0f78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:23.238564 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:23.238382 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:23.238564 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.238423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:23.238564 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:23.238467 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:24.238432485 +0000 UTC m=+40.215010171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:23.238564 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:23.238483 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:11:23.238564 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:23.238551 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:24.238536106 +0000 UTC m=+40.215113783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : configmap "kube-rbac-proxy" not found Oct 30 21:11:23.300043 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.300016 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbvl\" (UniqueName: \"kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl\") pod \"migrator-66df5d496b-n85d4\" (UID: \"b88c250c-3bfc-4b74-a0b3-b02be06b0f78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:23.582163 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.582084 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:23.772047 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.772010 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k67dh" event={"ID":"50585253-3ba0-43a4-a7ca-c38bec0bb1c6","Type":"ContainerStarted","Data":"9de7b2da74045d34dad783e606532659feee3e11ba2094e7fba0570fbc71209b"} Oct 30 21:11:23.772506 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:23.772076 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:24.014396 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:24.014343 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k67dh" podStartSLOduration=34.884463811 podStartE2EDuration="39.014304317s" totalImagesPullingTime="4.129840506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:23.898230232 +0000 UTC m=+39.874807924" watchObservedRunningTime="2026-10-30 21:11:24.014304317 +0000 UTC m=+39.990882008" Oct 30 21:11:24.015623 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:24.015514 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4"] Oct 30 21:11:24.018447 ip-10-0-137-220 kubenswrapper[2572]: W1030 21:11:24.018418 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c250c_3bfc_4b74_a0b3_b02be06b0f78.slice/crio-2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef WatchSource:0}: Error finding container 2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef: Status 404 returned error can't find the container with id 2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef Oct 30 21:11:24.247396 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:24.247359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:24.247589 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:24.247491 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:11:24.247589 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:24.247569 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:26.247550631 +0000 UTC m=+42.224128316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : configmap "kube-rbac-proxy" not found Oct 30 21:11:24.247713 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:24.247612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:24.247794 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:24.247779 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:24.247850 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:24.247836 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:26.247820565 +0000 UTC m=+42.224398252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:24.776943 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:24.776906 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" event={"ID":"b88c250c-3bfc-4b74-a0b3-b02be06b0f78","Type":"ContainerStarted","Data":"2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef"} Oct 30 21:11:25.346208 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:25.346169 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:25.346397 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:25.346237 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:25.346676 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:25.346656 2572 scope.go:122] "RemoveContainer" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" Oct 30 21:11:25.346879 ip-10-0-137-220 kubenswrapper[2572]: E1030 21:11:25.346849 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:25.781907 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:25.781874 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mp2j2" event={"ID":"503a46bf-3696-4a30-953b-19ed465b5919","Type":"ContainerStarted","Data":"f7ab0fbb65e38deef41039d3ff51d398f1134853972ed6fc67c0672fd7f49702"} Oct 30 21:11:25.901219 ip-10-0-137-220 kubenswrapper[2572]: I1030 21:11:25.901164 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mp2j2" podStartSLOduration=34.254371174 podStartE2EDuration="40.901150015s" totalImagesPullingTime="6.646778841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:25.899782336 +0000 UTC m=+41.876360028" watchObservedRunningTime="2026-10-30 21:11:25.901150015 +0000 UTC m=+41.877727703" Oct 30 21:11:25.949999 ip-10-0-137-220 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:11:25.964745 ip-10-0-137-220 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:11:25.964930 ip-10-0-137-220 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:11:25.965120 ip-10-0-137-220 systemd[1]: kubelet.service: Consumed 2.975s CPU time. Oct 30 21:11:25.979027 ip-10-0-137-220 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:11:26.072282 ip-10-0-137-220 kubenswrapper[4921]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:11:26.072282 ip-10-0-137-220 kubenswrapper[4921]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:11:26.072282 ip-10-0-137-220 kubenswrapper[4921]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:11:26.072282 ip-10-0-137-220 kubenswrapper[4921]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:11:26.074465 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074390 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.074465 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074465 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074470 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074475 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074479 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074482 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074484 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074487 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074490 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074492 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074495 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074497 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074500 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074502 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074505 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074507 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074510 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074513 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074515 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074518 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.074545 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074521 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074524 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074527 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074529 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074532 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074535 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074537 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074540 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074552 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074556 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074558 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074561 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074564 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074567 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074570 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074572 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074575 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074577 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074579 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074582 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.075025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074585 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074588 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074590 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074592 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074595 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074597 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074600 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074603 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074606 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074608 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074611 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074614 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074617 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074619 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074622 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074625 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074628 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074630 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074633 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.075528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074635 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074639 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074643 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074645 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074648 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074650 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074653 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074656 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074658 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074661 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074664 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074666 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074669 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074672 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074675 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074677 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074680 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074683 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074685 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074689 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.076012 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074691 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074694 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074696 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074699 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074702 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074704 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074707 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074710 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074712 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074715 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074717 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074720 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074724 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074727 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074731 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074734 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074739 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074741 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074744 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.076651 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074747 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074749 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074752 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074755 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074758 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074760 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074762 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.074765 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075730 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075739 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075742 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075745 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075748 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075750 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075753 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075756 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075759 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075762 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075764 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075767 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.077474 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075769 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075772 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075774 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075777 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075779 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075782 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075784 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075787 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075789 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075792 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075795 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075798 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075801 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075803 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075806 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075809 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075811 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075814 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075816 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075819 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.078039 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075822 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075824 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075828 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075833 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075836 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075839 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075842 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075845 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075848 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075850 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075853 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075855 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075857 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075860 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075862 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075865 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075868 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075871 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075874 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.078597 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075876 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075879 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075882 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075885 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075888 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075890 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075893 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075895 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075898 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075900 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075903 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075905 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075907 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075911 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075915 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075917 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075921 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075923 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075926 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075929 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.079087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075931 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075934 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075937 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075940 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075942 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075945 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075948 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075950 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075953 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075956 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075959 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075961 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075964 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075966 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075969 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075972 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075975 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075977 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075980 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075982 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.079811 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075985 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075987 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075990 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075993 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075996 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.075998 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076001 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076003 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076006 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076008 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076011 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076013 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076016 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076019 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.076021 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077011 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077033 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077038 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077042 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.080339 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077047 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077052 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077058 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077062 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077067 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077072 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077076 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077081 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077085 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077094 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077099 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077103 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077108 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077112 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077117 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077121 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077126 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077130 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077135 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077139 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.080828 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077143 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077148 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077157 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077162 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077166 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077170 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077175 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077180 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077184 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077190 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077195 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077199 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077204 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077208 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077213 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077222 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077227 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077231 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077236 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.081346 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077240 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077244 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077249 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077253 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077258 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077262 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077267 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077271 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077280 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077285 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077290 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077294 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077298 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077303 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077307 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077334 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077339 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077344 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077348 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077359 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.082156 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077363 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077373 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077378 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077383 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077388 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077392 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077396 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077400 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077405 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077409 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077414 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077418 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077422 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077427 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077436 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077440 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077444 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077449 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077453 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.083045 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077458 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077467 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077474 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077480 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077486 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077490 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077495 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077505 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077510 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077514 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077518 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077522 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077527 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077532 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077536 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077540 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077552 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077558 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077589 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.083921 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077618 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077744 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077762 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077768 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.077773 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077915 4921 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077931 4921 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077945 4921 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077954 4921 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077962 4921 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077968 4921 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077977 4921 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077983 4921 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077988 4921 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077991 4921 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077995 4921 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.077999 4921 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078002 4921 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078005 4921 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078008 4921 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078012 4921 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078015 4921 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078018 4921 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:11:26.084815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078024 4921 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078027 4921 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078030 4921 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078033 4921 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078037 4921 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078041 4921 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078044 4921 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078048 4921 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078052 4921 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078056 4921 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078060 4921 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078063 4921 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078066 4921 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078069 4921 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078074 4921 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078077 4921 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078080 4921 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078083 4921 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078086 4921 flags.go:64] FLAG: --enable-server="true" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078089 4921 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078094 4921 flags.go:64] FLAG: --event-burst="100" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078097 4921 flags.go:64] FLAG: --event-qps="50" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078100 4921 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078103 4921 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:11:26.085868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078106 4921 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078110 4921 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078113 4921 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078116 4921 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078120 4921 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078123 4921 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078127 4921 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078130 4921 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078133 4921 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078135 4921 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078139 4921 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078142 4921 flags.go:64] FLAG: --feature-gates="" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078151 4921 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078155 4921 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078158 4921 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078161 4921 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078166 4921 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078169 4921 flags.go:64] FLAG: --help="false" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078172 4921 flags.go:64] FLAG: --hostname-override="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078176 4921 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078179 4921 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078182 4921 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078186 4921 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:11:26.087000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078189 4921 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078192 4921 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078195 4921 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078198 4921 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078201 4921 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078204 4921 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078207 4921 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078210 4921 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078213 4921 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078216 4921 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078219 4921 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078222 4921 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078225 4921 flags.go:64] FLAG: --lock-file="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078228 4921 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078231 4921 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078234 4921 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078240 4921 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078243 4921 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078246 4921 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078249 4921 flags.go:64] FLAG: --logging-format="text" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078252 4921 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078255 4921 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078258 4921 flags.go:64] FLAG: --manifest-url="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078261 4921 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078270 4921 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:11:26.088083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078274 4921 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078278 4921 flags.go:64] FLAG: --max-pods="110" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078281 4921 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078284 4921 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078287 4921 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078290 4921 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078293 4921 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078296 4921 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078299 4921 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078306 4921 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078309 4921 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078332 4921 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078335 4921 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078338 4921 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078341 4921 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078344 4921 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078347 4921 flags.go:64] FLAG: --port="10250" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078350 4921 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078353 4921 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ded1d443e8932121" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078357 4921 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078360 4921 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078363 4921 flags.go:64] FLAG: --register-node="true" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078366 4921 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078370 4921 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078372 4921 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078375 4921 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:11:26.089176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078378 4921 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078382 4921 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078387 4921 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078390 4921 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078393 4921 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078396 4921 flags.go:64] FLAG: --runonce="false" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078401 4921 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078404 4921 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078407 4921 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078410 4921 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078412 4921 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078416 4921 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078419 4921 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078422 4921 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078425 4921 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078428 4921 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078431 4921 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078434 4921 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078437 4921 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078440 4921 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078443 4921 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078449 4921 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078452 4921 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078455 4921 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078460 4921 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:11:26.090294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078462 4921 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078466 4921 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078469 4921 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078472 4921 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078475 4921 flags.go:64] FLAG: --v="2" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078479 4921 flags.go:64] FLAG: --version="false" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078484 4921 flags.go:64] FLAG: --vmodule="" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078487 4921 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078490 4921 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078634 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078639 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078643 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078646 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078649 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078652 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078654 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078657 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078660 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078663 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078665 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078668 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078670 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.091091 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078673 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078675 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078678 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078680 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078683 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078686 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078689 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078692 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078694 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078697 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078700 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078702 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078706 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078708 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078711 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078713 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078716 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078719 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078721 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078725 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.091904 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078728 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078730 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078733 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078735 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078739 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078743 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078746 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078749 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078751 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078754 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078757 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078759 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078762 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078764 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078767 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078769 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078772 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078774 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078777 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.092533 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078780 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078782 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078785 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078787 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078790 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078792 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078795 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078798 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078800 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078803 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078805 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078808 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078812 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078814 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078817 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078819 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078822 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078824 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078827 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.092998 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078831 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078835 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078838 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078841 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078844 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078846 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078849 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078851 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078854 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078857 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078860 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078862 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078865 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078868 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078870 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078873 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078875 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078878 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078880 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.093616 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078882 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078886 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078889 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078891 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078894 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078896 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078901 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078903 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078906 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078908 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078911 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078914 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078916 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078919 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078921 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078924 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.094342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.078953 4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078989 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078993 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078996 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.078998 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079001 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079004 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079006 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079009 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079011 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079015 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079018 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079021 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079024 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079026 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079029 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079032 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079034 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079037 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079040 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.094771 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079043 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079045 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079048 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079052 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079055 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079057 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079060 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079063 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079065 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079068 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079070 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079073 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079077 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079081 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079083 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079086 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079089 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079092 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079095 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.095265 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079098 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079101 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079103 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079107 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079110 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079113 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079115 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079118 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079121 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079123 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079126 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079128 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079131 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079134 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079137 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079139 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079143 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079145 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079148 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079151 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.095749 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079153 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079156 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079158 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079161 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079163 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079166 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079169 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079173 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079176 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079179 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079183 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079185 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079188 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079190 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079193 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079196 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079199 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079202 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079205 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.096262 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079207 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079210 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079212 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079215 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079217 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079220 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079223 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079225 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079228 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079232 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079234 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079237 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079242 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079245 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079247 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079250 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079252 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079255 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079257 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079260 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.097150 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079263 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079265 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079268 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079271 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079274 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079277 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079279 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079282 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.079286 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.083804 4921 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.083824 4921 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.083851 4921 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.083860 4921 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083945 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083955 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083961 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083966 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083971 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083977 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.097716 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083984 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083989 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083994 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.083999 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084004 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084009 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084013 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084018 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084023 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084027 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084031 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084036 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084040 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084045 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084049 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084053 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084058 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084063 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084067 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.098185 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084071 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084076 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084080 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084085 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084091 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084096 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084101 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084105 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084110 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084115 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084119 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084123 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084127 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084132 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084136 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084141 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084145 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084149 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084153 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.098675 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084158 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084162 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084166 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084170 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084175 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084180 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084185 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084190 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084194 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084198 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084203 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084207 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084211 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084215 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084220 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084225 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084229 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084234 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084239 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084243 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.099163 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084247 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084252 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084256 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084260 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084265 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084269 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084273 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084277 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084282 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084286 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084290 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084294 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084298 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084303 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084307 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084311 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084331 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084336 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084341 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084345 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.099664 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084350 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084354 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084358 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084363 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084367 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084371 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084375 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084380 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084385 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084389 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084393 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084398 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084402 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084406 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084411 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084416 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084420 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084424 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084428 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084432 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.100146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084437 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084441 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.084482 4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084717 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084727 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084734 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084739 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084744 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084749 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084755 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084760 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084765 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084769 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084773 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084778 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084783 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.100658 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084787 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084791 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084795 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084800 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084804 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084808 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084812 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084816 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084822 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084826 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084830 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084834 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084838 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084842 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084847 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084851 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084855 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084860 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084864 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.101052 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084869 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084873 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084877 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084882 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084887 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084893 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084900 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084905 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084913 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084920 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084924 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084929 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084933 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084938 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084942 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084948 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084952 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084956 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084961 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.101599 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084965 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084969 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084973 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084978 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084983 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084987 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084992 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.084996 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085000 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085005 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085009 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085014 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085018 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085023 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085027 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085031 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085035 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085040 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085044 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085049 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.102078 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085053 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085059 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085063 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085068 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085072 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085076 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085081 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085085 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085090 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085094 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085098 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085104 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085110 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085115 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085120 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085124 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085129 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085134 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085138 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085142 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.102576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085146 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085150 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085154 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085159 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085163 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085168 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085172 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085176 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085181 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085185 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085189 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085193 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085198 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085204 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085208 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.085245 4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:11:26.103146 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085306 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085329 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085335 4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085339 4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085343 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085349 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085355 4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085363 4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085369 4921 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085375 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085380 4921 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085385 4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085390 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085395 4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085399 4921 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085404 4921 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085409 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085413 4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085417 4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:11:26.103564 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085421 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085426 4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085430 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085434 4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085439 4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085443 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085447 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085452 4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085456 4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085460 4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085465 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085471 4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085476 4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085480 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085485 4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085489 4921 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085493 4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085497 4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085502 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085508 4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:11:26.104043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085512 4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085516 4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085521 4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085525 4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085530 4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085534 4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085539 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085543 4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085548 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085553 4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085558 4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085562 4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085566 4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085572 4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085578 4921 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085583 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085588 4921 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085592 4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085596 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:11:26.104592 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085601 4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085605 4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085609 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085613 4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085620 4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085624 4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085629 4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085633 4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085637 4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085641 4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085645 4921 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085650 4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085656 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085660 4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085664 4921 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085668 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085673 4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085677 4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085682 4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085686 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:11:26.105071 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085690 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085696 4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085700 4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085705 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085710 4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085714 4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085718 4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085722 4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085726 4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085731 4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085735 4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085739 4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085743 4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085748 4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085752 4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085757 4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085762 4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085767 4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085771 4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085776 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:11:26.105657 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085780 4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085786 4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085790 4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085794 4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085801 4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085805 4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085809 4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:26.085814 4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.085893 4921 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087378 4921 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087463 4921 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087724 4921 server.go:1014] "Starting client certificate rotation" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087850 4921 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087892 4921 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:05:44 +0000 UTC" deadline="2028-05-02 04:51:26.257986518 +0000 UTC" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.087929 4921 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13183h40m0.170059352s" Oct 30 21:11:26.106180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.088435 4921 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:11:26.106589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.090044 4921 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:11:26.106589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.091511 4921 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:11:26.106589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.093267 4921 log.go:25] "Validated CRI v1 image API" Oct 30 21:11:26.106589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.093999 4921 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:11:26.106589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.096508 4921 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 83cc390b-e4e1-4423-9483-88e1eba3af4c:/dev/nvme0n1p4 a848376f-3c81-4d8a-a5da-3b3c510c9ed4:/dev/nvme0n1p3] Oct 30 21:11:26.106762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.096555 4921 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dbddf6eea81b44225ac548bf6186292f060916754fe7de8ef512076e7867093/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dbddf6eea81b44225ac548bf6186292f060916754fe7de8ef512076e7867093/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0620bbf3-fe2f-4967-b303-810f585e75e5/volumes/kubernetes.io~projected/kube-api-access-q2n2k:{mountpoint:/var/lib/kubelet/pods/0620bbf3-fe2f-4967-b303-810f585e75e5/volumes/kubernetes.io~projected/kube-api-access-q2n2k major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~projected/kube-api-access-l2tqj:{mountpoint:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~projected/kube-api-access-l2tqj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~projected/kube-api-access-8ns9z:{mountpoint:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~projected/kube-api-access-8ns9z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16ebb912-daf3-4d1c-b78c-46a14632cca5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/16ebb912-daf3-4d1c-b78c-46a14632cca5/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a68f610-9f32-485e-8814-8e624d1c1538/volumes/kubernetes.io~projected/kube-api-access-rl8g8:{mountpoint:/var/lib/kubelet/pods/1a68f610-9f32-485e-8814-8e624d1c1538/volumes/kubernetes.io~projected/kube-api-access-rl8g8 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26cead8f-43b1-4117-a777-2a571d56a16a/volumes/kubernetes.io~projected/kube-api-access-7ghnh:{mountpoint:/var/lib/kubelet/pods/26cead8f-43b1-4117-a777-2a571d56a16a/volumes/kubernetes.io~projected/kube-api-access-7ghnh major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/kube-api-access-cxw2t:{mountpoint:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/kube-api-access-cxw2t major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~projected/kube-api-access-h2d69:{mountpoint:/var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~projected/kube-api-access-h2d69 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/503a46bf-3696-4a30-953b-19ed465b5919/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/503a46bf-3696-4a30-953b-19ed465b5919/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50585253-3ba0-43a4-a7ca-c38bec0bb1c6/volumes/kubernetes.io~projected/kube-api-access-sc6w7:{mountpoint:/var/lib/kubelet/pods/50585253-3ba0-43a4-a7ca-c38bec0bb1c6/volumes/kubernetes.io~projected/kube-api-access-sc6w7 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~projected/kube-api-access-6c2g7:{mountpoint:/var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~projected/kube-api-access-6c2g7 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59bc969b-f10f-4b5b-a52c-788fda6582e4/volumes/kubernetes.io~projected/kube-api-access-mkfkj:{mountpoint:/var/lib/kubelet/pods/59bc969b-f10f-4b5b-a52c-788fda6582e4/volumes/kubernetes.io~projected/kube-api-access-mkfkj major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d316c29-1798-4c5a-9c1a-b8036fc6ccef/volumes/kubernetes.io~projected/kube-api-access-9s7wt:{mountpoint:/var/lib/kubelet/pods/5d316c29-1798-4c5a-9c1a-b8036fc6ccef/volumes/kubernetes.io~projected/kube-api-access-9s7wt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b/volumes/kubernetes.io~projected/kube-api-access-qhvtv:{mountpoint:/var/lib/kubelet/pods/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b/volumes/kubernetes.io~projected/kube-api-access-qhvtv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~projected/kube-api-access-2rcp5:{mountpoint:/var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~projected/kube-api-access-2rcp5 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~projected/kube-api-access-7rf6n:{mountpoint:/var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~projected/kube-api-access-7rf6n major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c440558-5078-4b4d-b28a-2c36109ab98e/volumes/kubernetes.io~projected/kube-api-access-ph4pp:{mountpoint:/var/lib/kubelet/pods/7c440558-5078-4b4d-b28a-2c36109ab98e/volumes/kubernetes.io~projected/kube-api-access-ph4pp major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85c676ae-424f-431f-b349-27eeb74c1c1a/volumes/kubernetes.io~projected/kube-api-access-8mdtc:{mountpoint:/var/lib/kubelet/pods/85c676ae-424f-431f-b349-27eeb74c1c1a/volumes/kubernetes.io~projected/kube-api-access-8mdtc major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~projected/kube-api-access-ttpt6:{mountpoint:/var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~projected/kube-api-access-ttpt6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7e01fb-ccea-4561-9de7-7810720748b7/volumes/kubernetes.io~projected/kube-api-access-cqgvp:{mountpoint:/var/lib/kubelet/pods/8e7e01fb-ccea-4561-9de7-7810720748b7/volumes/kubernetes.io~projected/kube-api-access-cqgvp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b110512e-ce89-4a9c-9183-daaf057807d5/volumes/kubernetes.io~projected/kube-api-access-5q59k:{mountpoint:/var/lib/kubelet/pods/b110512e-ce89-4a9c-9183-daaf057807d5/volumes/kubernetes.io~projected/kube-api-access-5q59k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75d3c73-4085-468b-ac88-0956a8351130/volumes/kubernetes.io~projected/kube-api-access-zkf9l:{mountpoint:/var/lib/kubelet/pods/b75d3c73-4085-468b-ac88-0956a8351130/volumes/kubernetes.io~projected/kube-api-access-zkf9l major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b88c250c-3bfc-4b74-a0b3-b02be06b0f78/volumes/kubernetes.io~projected/kube-api-access-sqbvl:{mountpoint:/var/lib/kubelet/pods/b88c250c-3bfc-4b74-a0b3-b02be06b0f78/volumes/kubernetes.io~projected/kube-api-access-sqbvl major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~projected/kube-api-access-rgrkq:{mountpoint:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~projected/kube-api-access-rgrkq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d120a1ef-dae4-4199-8149-0ae9660d8dfe/volumes/kubernetes.io~projected/kube-api-access-gmm6d:{mountpoint:/var/lib/kubelet/pods/d120a1ef-dae4-4199-8149-0ae9660d8dfe/volumes/kubernetes.io~projected/kube-api-access-gmm6d major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/kube-api-access-msqkl:{mountpoint:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/kube-api-access-msqkl major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc378851-2cf8-4b37-9ea4-efa2d9027272/volumes/kubernetes.io~projected/kube-api-access-qnhwx:{mountpoint:/var/lib/kubelet/pods/dc378851-2cf8-4b37-9ea4-efa2d9027272/volumes/kubernetes.io~projected/kube-api-access-qnhwx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99/volumes/kubernetes.io~projected/kube-api-access-7t56r:{mountpoint:/var/lib/kubelet/pods/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99/volumes/kubernetes.io~projected/kube-api-access-7t56r major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9686531-8650-4b17-a9f8-002a96ba1fb4/volumes/kubernetes.io~projected/kube-api-access-4frqj:{mountpoint:/var/lib/kubelet/pods/e9686531-8650-4b17-a9f8-002a96ba1fb4/volumes/kubernetes.io~projected/kube-api-access-4frqj major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efcd9b7e-a9d6-49c1-8792-69fec47961ee/volumes/kubernetes.io~projected/kube-api-access-wnwzx:{mountpoint:/var/lib/kubelet/pods/efcd9b7e-a9d6-49c1-8792-69fec47961ee/volumes/kubernetes.io~projected/kube-api-access-wnwzx major:0 minor:212 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4b5be7006e16f686ff60245e374119fecfeb24c05d450bfeb9f7fec927951bb8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fbd1135cb0671524c762fbdcaa8158e50f0dfd0ace57d1d1f25ca6fd3b41e510/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c002cf05c855491ac250d7a14ba126162adb29a4694a4c48489a4aa3a094d367/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/cd29e4acb56dffd20776775f18e6e06672b0835b4649127a0bb07b264bfd7d9e/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d3f272fc21c8182025886f1799cefcc95e6e58631071721a2b773eb437c34c00/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3a03cfaef663a290464ce13fde25e6e5b8296a8bbfccd923ed19cab6b04f5a6a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/87101506ddf96346a3d6f4be15020463085a27c481739608dc77b6667b4b1c31/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/99c2a73a21b33fd87793521f2d15b5be3baeb66a9c761d9f2bf955c6ef7ea13b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/b69b03c2d3994aea69b50769e7e13de7efc6dcd1f614dce872a5b9e669002242/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/93274f24ac2169b8c2de40759b3f9bc5f1cb0c2945be4851e7eb716700f1619d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/335f0d9eb1789adcea57db253224b5a8c8b14b390d77e8de20abe297623059ed/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c044ba42de598cc23a71d9725dcd59f0dc03f87e9effcbd58dd0bf257e21a517/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/eddc9812ec579dc083c21c02ef9fce28caed24c4965d26adf689a99926f4a762/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/eb67b892a91b5f1974c96cf047b81b89117bb79cd074f72dc38006f54d64220f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/261ff51d31a7536263ce5e36c5973ab1d37b8fb25d2b8e17b3f2ce6edb96ecc8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/218d81b5b61d8bcc7898f9252f0a43c80777d967fe521473358a8c54a87bf2e3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/21e7d3f62180fbb09e2d61a92dc5925202309675b483c365dcca2a593e263d0d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8c14dc8b0bef5fc90ef9dc4e0c31b128d929884a8dc2a3bcf3ba5c2d1a59131b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/26e7ef1b47b29947053aa2950f5a0422e59f26e5bec507a98b1917350a98448e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/77831f905759c9b2e1163780662bce96a4fae129d495775281bec8f107a55edb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f4c7c807fa7659e1f8a0cb3f19bdbc00f6115697298907f5aa1999f55bc93b2c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4cfe5341cf286735096b646f2c1f9058cfd3e564ccb554961cdc8191ea622121/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/08ac36d81e92cb63f0d9de5ae8c285f12d09511355ae91a70dae42f4b9dcced0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/61800547e5b3320570742143af3e0a08be64dd52618c31383ebe1f1819db92d0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a6a530b33180361c9165ceff9044ce984b88d3bfe5a6c9e8f1c564cca9f298c4/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/c719239787406212721c3ce42a4fe7fa251ec3302e3ff6868a271a0a0936bb96/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/926ef4ff814bb22928a30ed7dece349ca5f6de9302ca2c80dbec13249c6e0942/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/80f622f6e30a5a630e26c5933f0b1975ae2bce5855a8d3f8624c098dc53a73bd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/069cf0819f27ee31e609ecb92f3a17275a7aa9bc97b2613552efbfd32e9f48b4/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/2c1ec82411eb3626c79e8c631442ffc9cb4a83f08980d64dbb9c09fbc1cc67cd/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/403eea8b539890da0f4a03f008cdd13270b0e902f247d5a5ffdda39d71109350/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/2db85c2207d669c90739bdb18c8d3969b1896382045927268d001a4a98161828/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/5dbf5dfc04a615f02404cdc588803f9e1f149d320806d48560565920c29ae1fb/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/b964a01b412c942a91d815dbbd1ed07d90b04d6e8a210649f97119167dc1176d/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/3e4668ebe225731c4a3149453e58d52f41787dd040e00eb381ae24110ddda725/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a8ce4db9c3a8dc7aa867d8d34cb5a618080f3edbdb11ce775903e8f09e9616e8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/dbeb8d5179d292fdefaca856c12d5ce81fdea030a94ce4c511242e12d5c1627c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0adae2a63a3ed82502c4bbdbd4799cf6acf58f9dd2253881cf1a76c2e161bfc7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/e4b4f6a96fdc2d01192cda38a017902aa2c9be970e5844c3ee04cb9115da9d75/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3cb951bd3e7a9217c49c234a537609f39fe95269882ba0f2b04786a6cc189888/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/33f722a2ccdfc3fd62ca4cc9a572c5c54770657d8448076ae3b483efe6d17c51/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/cccfdfcde6c35f6a77fdd7a9351fd0ef63a7c05907a9b674c3cd2b6c13d40e55/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/663c63a861f81dd9484f158c769a2fb690bf62d61882bc05dcd7217eaa84450e/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/271a62d87078ce61d89e2688fba7b14039aac77130e3fcc1619cd32c60cbc819/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/331f01d0305d85cbd75666e2ffbd186c34e457c0011075000210bc0e5dc77080/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c38abaff843ac3f8b56442667157f6455113f311067f09fe7559d1fb964d72dd/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/45ca1fa68fb98b0068e5ab1ac2a842536760d965cfb109a3a73ccab612068e7b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2725d7b18e837bc110837cd7757fa20770ceb5e74ad1f7caa883d28f99a25d37/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ca43c1467682b33fec97229f9ced262e6673900b690baa8ee3ab984590fad8b1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1c3051031537b6d6f250fc7106c41b41793c9d59343a4c800e9d8e309cd4a9eb/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/7867b7aee6aeec4c4033d0d3fe7c079b5ab2071d8cb3e58e9ef63e24b535dcbd/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/20ff582547baac0082f0f7458122efbd141448b098389a1cc817cfdd347ba570/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e3323c6969a5531e95512c6acc35c6480aea044c6679132ac6c878d91ac8fa9c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/eca1f7d1732a18c4ec680addb3041b16452afd63e91d1fbed4acde9c4989a04e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/cd3c16d030da3c2cf878da53a80689361c0466f067e255ae2de7525613d29e07/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.105229 4921 manager.go:217] Machine: {Timestamp:2026-10-30 21:11:26.104776207 +0000 UTC m=+0.087476617 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099745 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b5677e61b0465162ea12b10f3208 SystemUUID:ec24b567-7e61-b046-5162-ea12b10f3208 BootID:1257dd80-9578-4519-bbca-358b2781137d Filesystems:[{Device:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26cead8f-43b1-4117-a777-2a571d56a16a/volumes/kubernetes.io~projected/kube-api-access-7ghnh DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc378851-2cf8-4b37-9ea4-efa2d9027272/volumes/kubernetes.io~projected/kube-api-access-qnhwx DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b75d3c73-4085-468b-ac88-0956a8351130/volumes/kubernetes.io~projected/kube-api-access-zkf9l DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~projected/kube-api-access-ttpt6 DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39c6bea9ff42451bddf7b45478286840c60359041a559a500b96e43d6a48d854/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/kube-api-access-cxw2t DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3e881fa336df036efa941195db799ca93b364004517f4cc3acce021bf68cbf0/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~projected/kube-api-access-h2d69 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85c676ae-424f-431f-b349-27eeb74c1c1a/volumes/kubernetes.io~projected/kube-api-access-8mdtc DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7652a639abb3d8c10ee5b2eefd196e6bc757b6cef79db3bc526d710c3b27e6bb/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d120a1ef-dae4-4199-8149-0ae9660d8dfe/volumes/kubernetes.io~projected/kube-api-access-gmm6d DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59bc969b-f10f-4b5b-a52c-788fda6582e4/volumes/kubernetes.io~projected/kube-api-access-mkfkj DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volumes/kubernetes.io~projected/kube-api-access-l2tqj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98a1b95bc81618237b3ace1eb3abfde8f85da14858f3e60c3b8a500618e7824b/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16ebb912-daf3-4d1c-b78c-46a14632cca5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13fabf842720508c71f282a39d00e3fbe4975813632da2b52ed022a5279667b6/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89e015bcb79f40c7e24af0b1f48233f2bb4220267aa20549b263302ffa50d87f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ad5662c6b76e9b7e1c0efa111014f292353ac727b7b2bd286394c71836882ec/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b110512e-ce89-4a9c-9183-daaf057807d5/volumes/kubernetes.io~projected/kube-api-access-5q59k DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f65ed91a9acaadcbfd5c3a75fcdddbec8c0d506fb1434d27f47fad29bfb63bf4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b2f8f25-7a5a-49fe-818c-67882dd9d616/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/volumes/kubernetes.io~projected/kube-api-access-rgrkq DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~projected/kube-api-access-msqkl DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/210ccf8449d335793987036f4ca89b2398040fad2095a9d7c956afddaf59dfb2/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37bc9be9e4e2a0f243a2cf7c0c6ca025794a35784b7acb47d9ff03eeee628eca/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9686531-8650-4b17-a9f8-002a96ba1fb4/volumes/kubernetes.io~projected/kube-api-access-4frqj DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f77b7c5525ff2a2f18c67d4829129206110c167ad95be68877899a00be957bd/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7c440558-5078-4b4d-b28a-2c36109ab98e/volumes/kubernetes.io~projected/kube-api-access-ph4pp DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/503a46bf-3696-4a30-953b-19ed465b5919/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e186ba90455488a2186893de54f444a60b9349ddf8f3c8be2461b94c4cc06f0d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45efd1e3739b852b58a80db656d3c22d7b6ec9b65e562741d87b1a2aac102880/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dbddf6eea81b44225ac548bf6186292f060916754fe7de8ef512076e7867093/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b88c250c-3bfc-4b74-a0b3-b02be06b0f78/volumes/kubernetes.io~projected/kube-api-access-sqbvl DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d97b7062330e6fdedcafc6d008688335508ad6b78eb06af6fffd20fd4b510ef/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0620bbf3-fe2f-4967-b303-810f585e75e5/volumes/kubernetes.io~projected/kube-api-access-q2n2k DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/volumes/kubernetes.io~projected/kube-api-access-7rf6n DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/143ba11974b3223c7f426589b474c2a776bf70aa654752e25e4414e36fc93d48/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abe5a1b6e663cff0824bb8dda3e7aaa16f6ce5ad8cf843e6813f91f1afd55676/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/711bdda1dd5fb215fa4ae4b1977a9f728c1ae64fca72d4708eb9a3f95d566c3c/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146/volumes/kubernetes.io~projected/kube-api-access-8ns9z DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efcd9b7e-a9d6-49c1-8792-69fec47961ee/volumes/kubernetes.io~projected/kube-api-access-wnwzx DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99/volumes/kubernetes.io~projected/kube-api-access-7t56r DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a68f610-9f32-485e-8814-8e624d1c1538/volumes/kubernetes.io~projected/kube-api-access-rl8g8 DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b/volumes/kubernetes.io~projected/kube-api-access-qhvtv DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/53efca0c-3152-434c-998e-d0916b982591/volumes/kubernetes.io~projected/kube-api-access-6c2g7 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f662559dacbfdae0035584f93aef98f31453003349b8b34708452e181136799/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d316c29-1798-4c5a-9c1a-b8036fc6ccef/volumes/kubernetes.io~projected/kube-api-access-9s7wt DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89bb38ba-2a52-40fc-9b88-a21200c1c542/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e37e4d1b20d70321081d1453cfabe6a598fa463faa9b2ef846013fbd6316e1ec/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7e01fb-ccea-4561-9de7-7810720748b7/volumes/kubernetes.io~projected/kube-api-access-cqgvp DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f0ca892-6436-4c1b-994c-bb97ba977307/volumes/kubernetes.io~projected/kube-api-access-2rcp5 DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50585253-3ba0-43a4-a7ca-c38bec0bb1c6/volumes/kubernetes.io~projected/kube-api-access-sc6w7 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b295f49-70ab-4d12-b08c-57f3dfb5324b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bb93d66a4ba6c2003ae0d2d209f5795d7a3cbdac7573b083cabe01f31764d94/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0bb93d66a4ba6c2 MacAddress:22:98:09:fc:d8:67 Speed:10000 Mtu:8901} {Name:13fabf842720508 MacAddress:9a:60:4e:09:dc:f3 Speed:10000 Mtu:8901} {Name:210ccf8449d3357 MacAddress:fe:68:26:6a:35:7c Speed:10000 Mtu:8901} {Name:2d97b7062330e6f MacAddress:a6:bc:f3:fa:be:cf Speed:10000 Mtu:8901} {Name:45efd1e3739b852 MacAddress:be:7e:60:a5:af:85 Speed:10000 Mtu:8901} {Name:6ad5662c6b76e9b MacAddress:ee:62:d3:0c:c6:bc Speed:10000 Mtu:8901} {Name:6dbddf6eea81b44 MacAddress:02:16:d9:44:9c:30 Speed:10000 Mtu:8901} {Name:711bdda1dd5fb21 MacAddress:6a:71:94:e3:9e:ca Speed:10000 Mtu:8901} {Name:7652a639abb3d8c MacAddress:62:47:73:38:3f:81 Speed:10000 Mtu:8901} {Name:89e015bcb79f40c MacAddress:a2:4c:92:52:54:33 Speed:10000 Mtu:8901} {Name:98a1b95bc816182 MacAddress:66:b6:ca:e1:12:ca Speed:10000 Mtu:8901} {Name:abe5a1b6e663cff MacAddress:2e:10:a4:7e:20:ce Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:41:39:11:9d:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:61:b7:25:dd:0a Speed:0 Mtu:8901} {Name:e37e4d1b20d7032 MacAddress:d6:70:3f:82:2c:b0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:41:39:11:9d:b1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:33:50:4b:79:9c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:21:59:91:92:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.105594 4921 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.105647 4921 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.105933 4921 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.105956 4921 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-220.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106082 4921 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106090 4921 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106100 4921 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106118 4921 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106175 4921 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106331 4921 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106441 4921 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106473 4921 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106490 4921 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106500 4921 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106508 4921 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:11:26.107005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.106516 4921 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:11:26.107621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.107350 4921 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:11:26.107621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.107360 4921 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:11:26.107810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.107794 4921 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:11:26.119985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.119968 4921 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:11:26.119985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.119984 4921 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:11:26.120248 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120232 4921 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:11:26.120562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120546 4921 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:11:26.120654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120589 4921 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:11:26.120724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120709 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120726 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120737 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120747 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120756 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120770 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:11:26.120786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120780 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120789 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120800 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120810 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120823 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120837 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120871 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:11:26.121109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.120881 4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:11:26.124600 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.124584 4921 server.go:1263] "Started kubelet" Oct 30 21:11:26.124735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.124683 4921 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:11:26.124735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.124677 4921 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:11:26.124828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.124753 4921 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:11:26.125103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.125045 4921 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:11:26.125092 ip-10-0-137-220 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:11:26.126020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.125967 4921 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:11:26.128424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.127871 4921 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:11:26.128424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.127936 4921 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:11:26.128424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.127988 4921 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:11:26.128424 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.128009 4921 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:11:26.128424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.128096 4921 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:11:26.128807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.128533 4921 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:11:26.128807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.128568 4921 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:05:46 +0000 UTC" deadline="2028-07-27 21:36:07.66600421 +0000 UTC" Oct 30 21:11:26.128807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.128597 4921 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15264h24m41.537410023s" Oct 30 21:11:26.130452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.130430 4921 factory.go:55] Registering systemd factory Oct 30 21:11:26.130452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.130448 4921 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:11:26.131302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.131281 4921 factory.go:153] Registering CRI-O factory Oct 30 21:11:26.131439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.131305 4921 factory.go:223] Registration of the crio container factory successfully Oct 30 21:11:26.131439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.131395 4921 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:11:26.131439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.131415 4921 factory.go:103] Registering Raw factory Oct 30 21:11:26.131439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.131431 4921 manager.go:1199] Started watching for new ooms in manager Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132434 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f8f25-7a5a-49fe-818c-67882dd9d616" volumeName="kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132499 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ebb912-daf3-4d1c-b78c-46a14632cca5" volumeName="kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132515 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c440558-5078-4b4d-b28a-2c36109ab98e" volumeName="kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132541 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a" volumeName="kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132555 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53efca0c-3152-434c-998e-d0916b982591" volumeName="kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132568 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b110512e-ce89-4a9c-9183-daaf057807d5" volumeName="kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132586 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132599 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c63ac6-3b39-4ee0-a5e1-1985c4d6b146" volumeName="kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132620 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59bc969b-f10f-4b5b-a52c-788fda6582e4" volumeName="kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132634 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" volumeName="kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132651 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132664 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc378851-2cf8-4b37-9ea4-efa2d9027272" volumeName="kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132677 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efcd9b7e-a9d6-49c1-8792-69fec47961ee" volumeName="kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132693 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0620bbf3-fe2f-4967-b303-810f585e75e5" volumeName="kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132710 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c63ac6-3b39-4ee0-a5e1-1985c4d6b146" volumeName="kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132730 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132743 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b295f49-70ab-4d12-b08c-57f3dfb5324b" volumeName="kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132761 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" volumeName="kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132773 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a68f610-9f32-485e-8814-8e624d1c1538" volumeName="kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132789 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26cead8f-43b1-4117-a777-2a571d56a16a" volumeName="kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132807 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2" volumeName="kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132821 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85c676ae-424f-431f-b349-27eeb74c1c1a" volumeName="kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132839 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a" volumeName="kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132853 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc7b666-f4e0-4a1d-8866-f7cd8b841f99" volumeName="kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132866 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132884 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b295f49-70ab-4d12-b08c-57f3dfb5324b" volumeName="kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132897 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2" volumeName="kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132918 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bb38ba-2a52-40fc-9b88-a21200c1c542" volumeName="kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132943 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc7b666-f4e0-4a1d-8866-f7cd8b841f99" volumeName="kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132957 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9686531-8650-4b17-a9f8-002a96ba1fb4" volumeName="kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132972 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ebb912-daf3-4d1c-b78c-46a14632cca5" volumeName="kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.132989 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133005 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133022 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133034 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f8f25-7a5a-49fe-818c-67882dd9d616" volumeName="kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133048 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f8f25-7a5a-49fe-818c-67882dd9d616" volumeName="kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133064 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f8f25-7a5a-49fe-818c-67882dd9d616" volumeName="kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133077 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133092 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133106 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b295f49-70ab-4d12-b08c-57f3dfb5324b" volumeName="kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133119 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133135 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133149 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50585253-3ba0-43a4-a7ca-c38bec0bb1c6" volumeName="kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133168 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53efca0c-3152-434c-998e-d0916b982591" volumeName="kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133181 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bb38ba-2a52-40fc-9b88-a21200c1c542" volumeName="kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133194 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75d3c73-4085-468b-ac88-0956a8351130" volumeName="kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133212 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133226 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d478b307-b6de-4b2a-b6cf-3a3b7bbbb414" volumeName="kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133243 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc378851-2cf8-4b37-9ea4-efa2d9027272" volumeName="kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133256 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d316c29-1798-4c5a-9c1a-b8036fc6ccef" volumeName="kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133269 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" volumeName="kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133289 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" volumeName="kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133302 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133333 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c440558-5078-4b4d-b28a-2c36109ab98e" volumeName="kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133347 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bb38ba-2a52-40fc-9b88-a21200c1c542" volumeName="kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133360 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e01fb-ccea-4561-9de7-7810720748b7" volumeName="kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133428 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a" volumeName="kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133447 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85c676ae-424f-431f-b349-27eeb74c1c1a" volumeName="kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133465 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b88c250c-3bfc-4b74-a0b3-b02be06b0f78" volumeName="kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133479 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d120a1ef-dae4-4199-8149-0ae9660d8dfe" volumeName="kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133493 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc378851-2cf8-4b37-9ea4-efa2d9027272" volumeName="kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133514 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133528 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f0ca892-6436-4c1b-994c-bb97ba977307" volumeName="kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133548 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133564 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15c63ac6-3b39-4ee0-a5e1-1985c4d6b146" volumeName="kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133579 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26cead8f-43b1-4117-a777-2a571d56a16a" volumeName="kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133599 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133614 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b295f49-70ab-4d12-b08c-57f3dfb5324b" volumeName="kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133630 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="503a46bf-3696-4a30-953b-19ed465b5919" volumeName="kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133646 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b110512e-ce89-4a9c-9183-daaf057807d5" volumeName="kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133658 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133668 4921 manager.go:319] Starting recovery of all containers Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133702 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc7b666-f4e0-4a1d-8866-f7cd8b841f99" volumeName="kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133717 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b2f8f25-7a5a-49fe-818c-67882dd9d616" volumeName="kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133734 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" volumeName="kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133748 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53efca0c-3152-434c-998e-d0916b982591" volumeName="kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133763 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59bc969b-f10f-4b5b-a52c-788fda6582e4" volumeName="kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133781 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3304c86-a470-4066-ace5-5927eb040db4" volumeName="kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133795 4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75d3c73-4085-468b-ac88-0956a8351130" volumeName="kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca" seLinuxMountContext="" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133808 4921 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:11:26.133973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.133818 4921 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:11:26.138255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.137101 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:26.139479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.139460 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:26.158554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.158519 4921 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:11:26.160226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.160199 4921 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:11:26.160364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.160232 4921 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:11:26.160364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.160262 4921 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:11:26.160364 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.160306 4921 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:11:26.174927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174889 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="d48d6a9e540453e3c155d5ffcc663a2e4b9e4b43542e44257d8bab1861ae97ed" exitCode=0 Oct 30 21:11:26.174927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174923 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="c8c928034bc6633cd00ff60b282c00a1473145f9459b050b785d7aff33ebd272" exitCode=0 Oct 30 21:11:26.174927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174932 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="25351818e5fb7ad055e9f1c78bef1a8fd1712c9500897a62a7b706d9c2a2aa74" exitCode=0 Oct 30 21:11:26.175135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174940 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="e98fa7e83fb0b791631aff4d413a05600f8d7d06238e0475c56197e28b5287e3" exitCode=0 Oct 30 21:11:26.175135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174947 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="63cec7598f201a3b63c7d1318fc5c4f4d289361488ed866ff970a43a0085485a" exitCode=0 Oct 30 21:11:26.175135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.174952 4921 generic.go:356] "Generic (PLEG): container finished" podID="5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b" containerID="7ed48fbb06e449dfcaed930871763b3c2d3d01bf2ed432e38146635d1cac024f" exitCode=0 Oct 30 21:11:26.180886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.180854 4921 generic.go:356] "Generic (PLEG): container finished" podID="6e8005a88bfac70fcc69b4dacc648a18" containerID="b816d015d6194f99d1872e56234c80201363e8d2e1516fd1b5770f0184490ac1" exitCode=0 Oct 30 21:11:26.203176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.203106 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:26.203176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.203133 4921 generic.go:356] "Generic (PLEG): container finished" podID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" exitCode=255 Oct 30 21:11:26.219719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.219693 4921 manager.go:324] Recovery completed Oct 30 21:11:26.228080 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.228051 4921 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:11:26.238598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.238577 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.239843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.239825 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.239928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.239856 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.239928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.239866 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.242406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242386 4921 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:11:26.242406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242404 4921 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:11:26.242538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242405 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.242538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242428 4921 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:11:26.242644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242585 4921 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:11:26.242644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242599 4921 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:11:26.242644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242634 4921 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:11:26.242644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242643 4921 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:11:26.242838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242652 4921 policy_none.go:50] "Start" Oct 30 21:11:26.242838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242665 4921 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:11:26.242838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242679 4921 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:11:26.242968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242867 4921 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:11:26.242968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242877 4921 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:11:26.242968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.242886 4921 policy_none.go:44] "Start" Oct 30 21:11:26.243331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.243303 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.243397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.243346 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.243397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.243361 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.247520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.247495 4921 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:11:26.250438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250425 4921 manager.go:341] "Starting Device Plugin manager" Oct 30 21:11:26.250501 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.250450 4921 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:11:26.250501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250459 4921 server.go:87] "Starting device plugin registration server" Oct 30 21:11:26.250778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250765 4921 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:11:26.250814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250781 4921 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:11:26.250887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250866 4921 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:11:26.251016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.250977 4921 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:11:26.251071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.251028 4921 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:11:26.251071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.251036 4921 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:11:26.251177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.251093 4921 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:11:26.251015416Z","UUID":"ada9422e-7058-4010-a7c3-663a236006c3","Handler":null,"Name":"","Endpoint":""} Oct 30 21:11:26.252764 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.252654 4921 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:11:26.252852 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.252818 4921 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" Oct 30 21:11:26.253260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.253241 4921 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:11:26.253367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.253352 4921 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:11:26.260465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.260425 4921 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal"] Oct 30 21:11:26.260572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.260559 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.261492 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.261474 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.261564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.261504 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.261564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.261517 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.261873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.261861 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.261987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.261975 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.262695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262678 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.262781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262700 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.262781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262708 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.262781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262721 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.262940 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262721 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.262940 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.262820 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.263012 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.262958 4921 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.263089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263050 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerStarted","Data":"2f688bdfe57323f50d9f3ac372065238fce106463dd482f3b76e7d94a3ae9e6b"} Oct 30 21:11:26.263125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263104 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerDied","Data":"b816d015d6194f99d1872e56234c80201363e8d2e1516fd1b5770f0184490ac1"} Oct 30 21:11:26.263153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263124 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.263220 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263128 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"6e8005a88bfac70fcc69b4dacc648a18","Type":"ContainerStarted","Data":"9f43efa718e88a2bb47c15bef221590d1762b6ad00d21670e4650cf90522f031"} Oct 30 21:11:26.263330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263239 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"48089fcbd21ef30454f56e9b574f5b67","Type":"ContainerStarted","Data":"a1547a971b0acd0397d9cbb851dff1d21a244ab9b0123af029aa0e46ecf438ab"} Oct 30 21:11:26.263330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263252 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" event={"ID":"48089fcbd21ef30454f56e9b574f5b67","Type":"ContainerStarted","Data":"8b29bdcab8959af9b07864cdbef0d5f270748f5da6d752698cf53261b3d80c0b"} Oct 30 21:11:26.263860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263847 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.263938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263875 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.263938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.263890 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.264054 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.264040 4921 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-220.us-west-1.compute.internal\" not found" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.314297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.314258 4921 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:11:26.314499 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.314381 4921 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:11:26.314499 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.314457 4921 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:11:26.315448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.315427 4921 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:11:26.328713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.328692 4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.350894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.350866 4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:11:26.351993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.351975 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:11:26.352100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.352010 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:11:26.352100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.352024 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:11:26.352100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.352076 4921 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.364943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.364911 4921 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:11:26.387996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.387964 4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:11:26.388110 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.388027 4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.388110 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.388042 4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.414598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.414567 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.414710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.414606 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.414710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.414632 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.459535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.459468 4921 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.459631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.459540 4921 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.460109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.460090 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:11:26.460159 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.460123 4921 setters.go:546] "Node became not ready" node="ip-10-0-137-220.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:11:26Z","lastTransitionTime":"2026-10-30T21:11:26Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:11:26.477133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.477113 4921 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-220.us-west-1.compute.internal" is owned by different node (oldNodeID="a4a0ec5f-3cfd-4e9f-aa85-c101534d5c88", newNodeID=""), cleaning up... Oct 30 21:11:26.489745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.489724 4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:11:26.489816 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:26.489769 4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515361 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515403 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515426 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515468 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515472 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e8005a88bfac70fcc69b4dacc648a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"6e8005a88bfac70fcc69b4dacc648a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:26.515663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:26.515473 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48089fcbd21ef30454f56e9b574f5b67-config\") pod \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" (UID: \"48089fcbd21ef30454f56e9b574f5b67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:27.107279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.107234 4921 apiserver.go:52] "Watching apiserver" Oct 30 21:11:27.216720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.216685 4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:27.216889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.216763 4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:27.269191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.269161 4921 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:11:27.269991 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.269965 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt","kube-system/kas-connection-checker-685895c7c9-44tx6","openshift-network-console/networking-console-plugin-86c478bb88-6bt4k","openshift-console-operator/console-operator-7cd56b8dbc-rvc5s","openshift-dns/dns-default-dspqr","openshift-network-operator/iptables-alerter-w7cdp","openshift-ingress/router-default-6dc5bbb4f7-hmkgf","openshift-image-registry/node-ca-bq4pk","openshift-network-diagnostics/network-check-target-k67dh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j","openshift-dns/node-resolver-hqlzb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-6d887","openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg","openshift-image-registry/image-registry-b9ddc8f6f-shs2w","openshift-ingress-canary/ingress-canary-qmnhw","openshift-insights/insights-runtime-extractor-76zdg","kube-system/global-pull-secret-syncer-mp2j2","openshift-insights/insights-operator-7c4986d449-rs27v","openshift-multus/network-metrics-daemon-xrkn6","kube-system/kas-connection-checker-685895c7c9-xmkfl","openshift-multus/multus-additional-cni-plugins-flz96","openshift-multus/multus-jt62z","openshift-cluster-node-tuning-operator/tuned-q4s6r","openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89","kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8","kube-system/konnectivity-agent-z2rbr","openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4","openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm","kube-system/kas-connection-checker-685895c7c9-4vfv6"] Oct 30 21:11:27.275388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.275364 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.276057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.276036 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.276680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.276661 4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:11:27.276783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.276703 4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:27.276783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.276752 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.277127 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.277110 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:27.277242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.277225 4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:11:27.277302 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.277266 4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-220.us-west-1.compute.internal" Oct 30 21:11:27.277487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.277473 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.280896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.280881 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:27.283259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.283244 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.284286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.284264 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.285221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.285207 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.322595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.322564 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.322595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.322588 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.325739 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325716 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:11:27.325868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325750 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.325868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325770 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.325868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325784 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.325969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325959 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:11:27.325998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.325971 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.326062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.326045 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.326620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.326599 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9dh6n\"" Oct 30 21:11:27.327081 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327067 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m4h2f\"" Oct 30 21:11:27.327257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327192 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.327257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327253 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:11:27.327382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327332 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:11:27.327443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327348 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.327443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327407 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.327443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327407 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.327590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327459 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:11:27.327590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327410 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:11:27.327590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327566 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.327846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327820 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:11:27.327899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327860 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:11:27.327899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327866 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-46nl2\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327904 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327941 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327942 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327962 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327944 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.327994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.327974 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:11:27.328215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.328018 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.328215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.328075 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:11:27.328215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.328079 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:11:27.328215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.328132 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.328215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.328178 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.332281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.332149 4921 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:11:27.371263 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.371190 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6vs6h\"" Oct 30 21:11:27.376282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.376259 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:11:27.376450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.376433 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:11:27.419684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419647 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.419684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419686 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.419896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419703 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.419896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419726 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.419896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419775 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.419995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.419972 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.420036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420002 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.420036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420000 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53efca0c-3152-434c-998e-d0916b982591-config\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.420093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420044 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.420093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420083 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.420157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420119 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2tqj\" (UniqueName: \"kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.420157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420138 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q59k\" (UniqueName: \"kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.420222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420163 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.420290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420270 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.420355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420301 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.420355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420336 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85c676ae-424f-431f-b349-27eeb74c1c1a-data-volume\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.420355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420346 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.420464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420372 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.420464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420412 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.420561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420463 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.420561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420542 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkf9l\" (UniqueName: \"kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.420663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420584 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:27.420663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420614 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.420663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420642 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2d69\" (UniqueName: \"kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.420767 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420670 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.420767 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420733 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.420830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420810 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.420882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420865 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.420922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420914 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2g7\" (UniqueName: \"kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.420957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420933 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.420957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420949 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.421045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420965 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.421045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420982 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2n2k\" (UniqueName: \"kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k\") pod \"kas-connection-checker-685895c7c9-xmkfl\" (UID: \"0620bbf3-fe2f-4967-b303-810f585e75e5\") " pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:27.421045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.420997 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwzx\" (UniqueName: \"kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx\") pod \"kas-connection-checker-685895c7c9-4vfv6\" (UID: \"efcd9b7e-a9d6-49c1-8792-69fec47961ee\") " pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:27.421045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421015 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.421045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421038 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421060 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghnh\" (UniqueName: \"kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421098 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t56r\" (UniqueName: \"kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421129 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421153 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421179 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421204 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421233 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frqj\" (UniqueName: \"kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj\") pod \"kas-connection-checker-685895c7c9-44tx6\" (UID: \"e9686531-8650-4b17-a9f8-002a96ba1fb4\") " pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:27.421290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421270 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421306 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-trusted-ca\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421333 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421368 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421399 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421460 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421477 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmm6d\" (UniqueName: \"kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d\") pod \"volume-data-source-validator-58c9f5f46d-6s6mw\" (UID: \"d120a1ef-dae4-4199-8149-0ae9660d8dfe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421497 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421511 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421527 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421542 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421561 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421618 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.421631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421619 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421668 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8g8\" (UniqueName: \"kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8\") pod \"network-check-source-6f78d59f99-rv9lg\" (UID: \"1a68f610-9f32-485e-8814-8e624d1c1538\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421700 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421722 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421739 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgvp\" (UniqueName: \"kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421763 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421794 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ns9z\" (UniqueName: \"kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421825 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421851 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.421907 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b295f49-70ab-4d12-b08c-57f3dfb5324b-config\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422008 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422039 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rf6n\" (UniqueName: \"kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:27.422095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422084 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422141 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422206 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422256 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4pp\" (UniqueName: \"kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422279 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422297 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422330 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422358 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422384 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422409 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422445 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.422495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422475 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b295f49-70ab-4d12-b08c-57f3dfb5324b-serving-cert\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422513 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422540 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422562 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422575 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-service-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422582 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422613 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422645 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422680 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7wt\" (UniqueName: \"kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422714 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422747 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-default-certificate\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.422787 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422787 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422807 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422825 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpt6\" (UniqueName: \"kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422846 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422869 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422884 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0ca892-6436-4c1b-994c-bb97ba977307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422893 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422920 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422959 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-tmp\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.422989 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/53efca0c-3152-434c-998e-d0916b982591-serving-cert\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423001 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhwx\" (UniqueName: \"kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.423175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423061 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423181 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423217 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423256 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423275 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423300 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423364 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423364 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423408 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423424 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423442 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423450 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423470 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423492 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423497 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvtv\" (UniqueName: \"kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423533 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423539 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423572 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423536 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423587 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423609 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423621 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423630 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423632 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423677 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.423666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423683 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423694 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423759 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbvl\" (UniqueName: \"kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl\") pod \"migrator-66df5d496b-n85d4\" (UID: \"b88c250c-3bfc-4b74-a0b3-b02be06b0f78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423796 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423860 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423872 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-config-volume\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423916 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423960 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.423995 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424027 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424056 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424063 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424068 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424075 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424084 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424089 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59bc969b-f10f-4b5b-a52c-788fda6582e4-tmp-dir\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424098 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424170 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89bb38ba-2a52-40fc-9b88-a21200c1c542-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424178 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-tmp\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424187 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424183 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424213 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424226 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424228 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424237 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424241 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424232 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424174 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424258 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfkj\" (UniqueName: \"kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424299 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424265 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424362 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424273 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424409 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424419 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7c440558-5078-4b4d-b28a-2c36109ab98e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424435 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424457 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424543 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424618 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424712 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424768 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424776 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424782 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.424825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424797 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424867 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-config\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424878 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424916 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424946 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrkq\" (UniqueName: \"kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.424949 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-stats-auth\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425028 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425050 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425093 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425176 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425205 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0ca892-6436-4c1b-994c-bb97ba977307-serving-cert\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425225 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425236 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5f0ca892-6436-4c1b-994c-bb97ba977307-snapshots\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425257 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425284 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425331 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425332 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425352 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcp5\" (UniqueName: \"kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425358 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-tmp-dir\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425388 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425494 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425535 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425576 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425600 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425614 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-tuned\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425634 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425664 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425690 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425714 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425736 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425756 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425779 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdtc\" (UniqueName: \"kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425802 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425876 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.425959 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89bb38ba-2a52-40fc-9b88-a21200c1c542-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.426079 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:11:27.426519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.426387 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427230 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427235 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb6bb\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427280 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427288 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z8gbk\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427294 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xkwpv\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427299 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427301 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2dlm\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427281 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427405 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-k8b2g\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427435 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427443 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8ztc5\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427451 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427501 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427530 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rvh9q\"" Oct 30 21:11:27.427695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427660 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427711 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dssbm\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427727 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4qh6\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427744 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427751 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tzzvv\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427762 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87wxx\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427778 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rvqfx\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427806 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mfb44\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427815 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427810 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wtj2v\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427753 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-swkpp\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427829 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9kft\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427836 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tklwz\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427828 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427852 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qm8dr\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427868 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwgx7\"" Oct 30 21:11:27.428155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.427862 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8s65\"" Oct 30 21:11:27.430356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.430340 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovn-node-metrics-cert\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.430397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.430343 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.430461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.430448 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b110512e-ce89-4a9c-9183-daaf057807d5-iptables-alerter-script\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.430717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.430702 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.430923 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.430909 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16ebb912-daf3-4d1c-b78c-46a14632cca5-konnectivity-ca\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:27.431067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431055 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-daemon-config\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.431209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431197 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.431383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431368 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.431445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431431 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-env-overrides\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.431564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431552 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16ebb912-daf3-4d1c-b78c-46a14632cca5-agent-certs\") pod \"konnectivity-agent-z2rbr\" (UID: \"16ebb912-daf3-4d1c-b78c-46a14632cca5\") " pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:27.431595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431567 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.431651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.431638 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc378851-2cf8-4b37-9ea4-efa2d9027272-cni-binary-copy\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.432512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.432495 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.432632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.432615 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:27.432689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.432673 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75d3c73-4085-468b-ac88-0956a8351130-serviceca\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.432811 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.432799 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-cert\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:27.432923 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.432911 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.433390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.433376 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.434501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.434486 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.434540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.434507 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/503a46bf-3696-4a30-953b-19ed465b5919-original-pull-secret\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.434610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.434597 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cni-binary-copy\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.434735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.434719 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.434775 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.434762 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b2f8f25-7a5a-49fe-818c-67882dd9d616-ovnkube-script-lib\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.476074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.476037 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2tqj\" (UniqueName: \"kubernetes.io/projected/0b2f8f25-7a5a-49fe-818c-67882dd9d616-kube-api-access-l2tqj\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.476217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.476035 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkf9l\" (UniqueName: \"kubernetes.io/projected/b75d3c73-4085-468b-ac88-0956a8351130-kube-api-access-zkf9l\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.479798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.479778 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.480485 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.480468 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2n2k\" (UniqueName: \"kubernetes.io/projected/0620bbf3-fe2f-4967-b303-810f585e75e5-kube-api-access-q2n2k\") pod \"kas-connection-checker-685895c7c9-xmkfl\" (UID: \"0620bbf3-fe2f-4967-b303-810f585e75e5\") " pod="kube-system/kas-connection-checker-685895c7c9-xmkfl" Oct 30 21:11:27.480544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.480483 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.480959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.480924 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4pp\" (UniqueName: \"kubernetes.io/projected/7c440558-5078-4b4d-b28a-2c36109ab98e-kube-api-access-ph4pp\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.481171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481139 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgvp\" (UniqueName: \"kubernetes.io/projected/8e7e01fb-ccea-4561-9de7-7810720748b7-kube-api-access-cqgvp\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.481327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481202 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8g8\" (UniqueName: \"kubernetes.io/projected/1a68f610-9f32-485e-8814-8e624d1c1538-kube-api-access-rl8g8\") pod \"network-check-source-6f78d59f99-rv9lg\" (UID: \"1a68f610-9f32-485e-8814-8e624d1c1538\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rv9lg" Oct 30 21:11:27.481327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481270 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q59k\" (UniqueName: \"kubernetes.io/projected/b110512e-ce89-4a9c-9183-daaf057807d5-kube-api-access-5q59k\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.481503 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481462 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ns9z\" (UniqueName: \"kubernetes.io/projected/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-kube-api-access-8ns9z\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.481679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481660 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdtc\" (UniqueName: \"kubernetes.io/projected/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-api-access-8mdtc\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.481743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481722 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7wt\" (UniqueName: \"kubernetes.io/projected/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-kube-api-access-9s7wt\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:27.481776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481745 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbvl\" (UniqueName: \"kubernetes.io/projected/b88c250c-3bfc-4b74-a0b3-b02be06b0f78-kube-api-access-sqbvl\") pod \"migrator-66df5d496b-n85d4\" (UID: \"b88c250c-3bfc-4b74-a0b3-b02be06b0f78\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" Oct 30 21:11:27.481897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481881 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhwx\" (UniqueName: \"kubernetes.io/projected/dc378851-2cf8-4b37-9ea4-efa2d9027272-kube-api-access-qnhwx\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.481939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.481901 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rf6n\" (UniqueName: \"kubernetes.io/projected/7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2-kube-api-access-7rf6n\") pod \"ingress-canary-qmnhw\" (UID: \"7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2\") " pod="openshift-ingress-canary/ingress-canary-qmnhw" Oct 30 21:11:27.482616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.482576 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghnh\" (UniqueName: \"kubernetes.io/projected/26cead8f-43b1-4117-a777-2a571d56a16a-kube-api-access-7ghnh\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.482616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.482609 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.483148 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483124 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpt6\" (UniqueName: \"kubernetes.io/projected/89bb38ba-2a52-40fc-9b88-a21200c1c542-kube-api-access-ttpt6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j\" (UID: \"89bb38ba-2a52-40fc-9b88-a21200c1c542\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" Oct 30 21:11:27.483309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483128 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t56r\" (UniqueName: \"kubernetes.io/projected/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-kube-api-access-7t56r\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.483309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483158 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmm6d\" (UniqueName: \"kubernetes.io/projected/d120a1ef-dae4-4199-8149-0ae9660d8dfe-kube-api-access-gmm6d\") pod \"volume-data-source-validator-58c9f5f46d-6s6mw\" (UID: \"d120a1ef-dae4-4199-8149-0ae9660d8dfe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6s6mw" Oct 30 21:11:27.483309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483301 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2d69\" (UniqueName: \"kubernetes.io/projected/4b295f49-70ab-4d12-b08c-57f3dfb5324b-kube-api-access-h2d69\") pod \"console-operator-7cd56b8dbc-rvc5s\" (UID: \"4b295f49-70ab-4d12-b08c-57f3dfb5324b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:27.483550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483329 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2g7\" (UniqueName: \"kubernetes.io/projected/53efca0c-3152-434c-998e-d0916b982591-kube-api-access-6c2g7\") pod \"service-ca-operator-544d749c8c-96rhm\" (UID: \"53efca0c-3152-434c-998e-d0916b982591\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" Oct 30 21:11:27.483550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483455 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrkq\" (UniqueName: \"kubernetes.io/projected/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-kube-api-access-rgrkq\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.483550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483500 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.484112 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwzx\" (UniqueName: \"kubernetes.io/projected/efcd9b7e-a9d6-49c1-8792-69fec47961ee-kube-api-access-wnwzx\") pod \"kas-connection-checker-685895c7c9-4vfv6\" (UID: \"efcd9b7e-a9d6-49c1-8792-69fec47961ee\") " pod="kube-system/kas-connection-checker-685895c7c9-4vfv6" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.483779 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frqj\" (UniqueName: \"kubernetes.io/projected/e9686531-8650-4b17-a9f8-002a96ba1fb4-kube-api-access-4frqj\") pod \"kas-connection-checker-685895c7c9-44tx6\" (UID: \"e9686531-8650-4b17-a9f8-002a96ba1fb4\") " pod="kube-system/kas-connection-checker-685895c7c9-44tx6" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.484144 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvtv\" (UniqueName: \"kubernetes.io/projected/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-kube-api-access-qhvtv\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.484334 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcp5\" (UniqueName: \"kubernetes.io/projected/5f0ca892-6436-4c1b-994c-bb97ba977307-kube-api-access-2rcp5\") pod \"insights-operator-7c4986d449-rs27v\" (UID: \"5f0ca892-6436-4c1b-994c-bb97ba977307\") " pod="openshift-insights/insights-operator-7c4986d449-rs27v" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.484374 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfkj\" (UniqueName: \"kubernetes.io/projected/59bc969b-f10f-4b5b-a52c-788fda6582e4-kube-api-access-mkfkj\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.487254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.484523 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6w7\" (UniqueName: \"kubernetes.io/projected/50585253-3ba0-43a4-a7ca-c38bec0bb1c6-kube-api-access-sc6w7\") pod \"network-check-target-k67dh\" (UID: \"50585253-3ba0-43a4-a7ca-c38bec0bb1c6\") " pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:27.526269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526218 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526266 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526284 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526303 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526341 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526347 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-netns\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526357 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-hostroot\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526366 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-etc-kubernetes\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526364 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526400 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b110512e-ce89-4a9c-9183-daaf057807d5-host-slash\") pod \"iptables-alerter-w7cdp\" (UID: \"b110512e-ce89-4a9c-9183-daaf057807d5\") " pod="openshift-network-operator/iptables-alerter-w7cdp" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526422 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-sys-fs\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526422 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-sys\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526460 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526498 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526516 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526549 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526564 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-multus\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526571 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526583 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-ovn\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526595 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-cni-bin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526596 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysconfig\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526610 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526613 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526625 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526644 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526670 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526702 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.026686043 +0000 UTC m=+2.009386440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526714 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526719 4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526727 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526734 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526759 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.026740978 +0000 UTC m=+2.009441376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526772 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-netns\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526782 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526807 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526825 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526825 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.526845 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.026828393 +0000 UTC m=+2.009528798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526842 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-run\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526855 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-socket-dir-parent\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526870 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-conf-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526876 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526906 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526930 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526937 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-netd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526962 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-device-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.526957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526961 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.526994 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527020 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527025 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-system-cni-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527034 4921 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527042 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-log-socket\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527052 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527081 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527088 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027076096 +0000 UTC m=+2.009776510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527107 4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527114 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527125 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-cnibin\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527136 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-socket-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527143 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527157 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027142979 +0000 UTC m=+2.009843390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527169 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-systemd-units\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527183 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-host\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527189 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527213 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527231 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527247 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-multus-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527248 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527282 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527293 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-systemd\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527305 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527328 4921 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527334 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527363 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527370 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027357703 +0000 UTC m=+2.010058115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527373 4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527395 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-kubelet\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527398 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527410 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027399304 +0000 UTC m=+2.010099735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527416 4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527434 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-slash\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527436 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527450 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027440925 +0000 UTC m=+2.010141321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527455 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-lib-modules\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527474 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-kubernetes\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527473 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527507 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527513 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-dbus\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527531 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527539 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-var-lib-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527559 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527582 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527608 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527621 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/503a46bf-3696-4a30-953b-19ed465b5919-kubelet-config\") pod \"global-pull-secret-syncer-mp2j2\" (UID: \"503a46bf-3696-4a30-953b-19ed465b5919\") " pod="kube-system/global-pull-secret-syncer-mp2j2" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527623 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flz96\" (UID: \"5e7b0ca6-fb24-4fd2-b428-fa3bd0d7553b\") " pod="openshift-multus/multus-additional-cni-plugins-flz96" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527632 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527657 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527671 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527649 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-cnibin\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527660 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-multus-certs\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527684 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-run-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527686 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527728 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-etc-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527734 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527763 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527784 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527800 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527816 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527824 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-modprobe-d\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527856 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-systemd\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527861 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85c676ae-424f-431f-b349-27eeb74c1c1a-crio-socket\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527859 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527877 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85c676ae-424f-431f-b349-27eeb74c1c1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527896 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527879 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-var-lib-kubelet\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527897 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527929 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.527942 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.027927766 +0000 UTC m=+2.010628166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527967 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527979 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59bc969b-f10f-4b5b-a52c-788fda6582e4-hosts-file\") pod \"node-resolver-hqlzb\" (UID: \"59bc969b-f10f-4b5b-a52c-788fda6582e4\") " pod="openshift-dns/node-resolver-hqlzb" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.527993 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528021 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528025 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7e01fb-ccea-4561-9de7-7810720748b7-registration-dir\") pod \"aws-ebs-csi-driver-node-ltnwt\" (UID: \"8e7e01fb-ccea-4561-9de7-7810720748b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltnwt" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528045 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528062 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-host-run-k8s-cni-cncf-io\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528074 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528097 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-run-openvswitch\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528099 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.528129 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.528143 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528142 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc378851-2cf8-4b37-9ea4-efa2d9027272-system-cni-dir\") pod \"multus-jt62z\" (UID: \"dc378851-2cf8-4b37-9ea4-efa2d9027272\") " pod="openshift-multus/multus-jt62z" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528157 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75d3c73-4085-468b-ac88-0956a8351130-host\") pod \"node-ca-bq4pk\" (UID: \"b75d3c73-4085-468b-ac88-0956a8351130\") " pod="openshift-image-registry/node-ca-bq4pk" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528158 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.528303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528130 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-node-log\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:27.528179 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:28.028166765 +0000 UTC m=+2.010867187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528215 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528242 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528268 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528305 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b2f8f25-7a5a-49fe-818c-67882dd9d616-host-cni-bin\") pod \"ovnkube-node-6d887\" (UID: \"0b2f8f25-7a5a-49fe-818c-67882dd9d616\") " pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528358 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-var-lib-kubelet\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.530832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.528402 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/15c63ac6-3b39-4ee0-a5e1-1985c4d6b146-etc-sysctl-conf\") pod \"tuned-q4s6r\" (UID: \"15c63ac6-3b39-4ee0-a5e1-1985c4d6b146\") " pod="openshift-cluster-node-tuning-operator/tuned-q4s6r" Oct 30 21:11:27.575353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.575305 4921 scope.go:122] "RemoveContainer" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" Oct 30 21:11:27.586807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:27.586785 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:11:28.032492 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032457 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032516 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032557 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032579 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032605 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032612 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032635 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032662 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:28.032694 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032684 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032707 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032723 4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032688 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032668059 +0000 UTC m=+3.015368476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032693 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032761 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032760 4921 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032721 4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032771 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032755424 +0000 UTC m=+3.015455851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032845 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032831304 +0000 UTC m=+3.015531716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032862 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032854703 +0000 UTC m=+3.015555101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032873 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032868558 +0000 UTC m=+3.015568956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032884 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032879194 +0000 UTC m=+3.015579591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.032900 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.032895046 +0000 UTC m=+3.015595442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032937 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032961 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:28.033037 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.032980 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033065 4921 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033081 4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033098 4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033115 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.033100354 +0000 UTC m=+3.015800757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033129 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.033121905 +0000 UTC m=+3.015822301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:28.033569 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.033140 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:29.033134974 +0000 UTC m=+3.015835370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:28.221734 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.221707 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/2.log" Oct 30 21:11:28.222172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.222156 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/1.log" Oct 30 21:11:28.222211 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.222188 4921 generic.go:356] "Generic (PLEG): container finished" podID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" exitCode=255 Oct 30 21:11:28.222287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.222265 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerDied","Data":"e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc"} Oct 30 21:11:28.222360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.222349 4921 scope.go:122] "RemoveContainer" containerID="137c5a433cb3b9d3319cd5d0a075e58b127c5433dc4451238f70398c9a8930e5" Oct 30 21:11:28.222722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.222707 4921 scope.go:122] "RemoveContainer" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" Oct 30 21:11:28.222876 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:28.222856 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:28.492660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:28.492632 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:29.040702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040670 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040710 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040755 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040776 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040797 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040820 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040839 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040840 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040870 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:29.040917 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040902 4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040927 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.040909291 +0000 UTC m=+5.023609689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040906 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040939 4921 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040955 4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040957 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040974 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040984 4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040999 4921 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040944 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.040902 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.040952 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.040934638 +0000 UTC m=+5.023635049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041076 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041060188 +0000 UTC m=+5.023760601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041097 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041087293 +0000 UTC m=+5.023787696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041125 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041116229 +0000 UTC m=+5.023816643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041139 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041131414 +0000 UTC m=+5.023831824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041155 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.04114659 +0000 UTC m=+5.023846988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041169 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041161235 +0000 UTC m=+5.023861637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041191 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.041184133 +0000 UTC m=+5.023884554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.041218 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:29.041360 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041337 4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:29.042050 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:29.041380 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:31.04136829 +0000 UTC m=+5.024068700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:29.227149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.227121 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/2.log" Oct 30 21:11:29.228688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.228660 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n85d4" event={"ID":"b88c250c-3bfc-4b74-a0b3-b02be06b0f78","Type":"ContainerStarted","Data":"378e900116c0bafdf21f39e688dd39700ece9d0d2753c09564d1d896f47d775d"} Oct 30 21:11:29.596527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:29.596502 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hqlzb_59bc969b-f10f-4b5b-a52c-788fda6582e4/dns-node-resolver/0.log" Oct 30 21:11:30.624526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:30.624488 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:30.625027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:30.624949 4921 scope.go:122] "RemoveContainer" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" Oct 30 21:11:30.625185 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:30.625163 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:30.796357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:30.796330 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq4pk_b75d3c73-4085-468b-ac88-0956a8351130/node-ca/0.log" Oct 30 21:11:30.973332 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:30.973294 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:30.998445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:30.998416 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:31.060612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060577 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060634 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060707 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060728 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060740 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060748 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060754 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:31.060783 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060774 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060761781 +0000 UTC m=+9.043462183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060801 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060809 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060797091 +0000 UTC m=+9.043497487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060847 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060855 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060846118 +0000 UTC m=+9.043546515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060862 4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060887 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060906 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060893422 +0000 UTC m=+9.043593835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060910 4921 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060930 4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060942 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060933669 +0000 UTC m=+9.043634066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.060953 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.060946898 +0000 UTC m=+9.043647295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.060987 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.061009 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.061031 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061061 4921 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061073 4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:31.061109 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061084 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.061077256 +0000 UTC m=+9.043777679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061143 4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061159 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.061115 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061179 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.061167691 +0000 UTC m=+9.043868089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061198 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.061188766 +0000 UTC m=+9.043889165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:31.061697 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.061214 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:35.061205797 +0000 UTC m=+9.043906194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:31.102804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.102773 4921 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:31.238626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.237525 4921 scope.go:122] "RemoveContainer" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" Oct 30 21:11:31.238626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.237599 4921 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:11:31.238626 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:31.237861 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:31.664310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.664225 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmnhw_7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/serve-healthcheck-canary/0.log" Oct 30 21:11:31.996152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:31.996126 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-n85d4_b88c250c-3bfc-4b74-a0b3-b02be06b0f78/migrator/0.log" Oct 30 21:11:32.197462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:32.197429 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j_89bb38ba-2a52-40fc-9b88-a21200c1c542/kube-storage-version-migrator-operator/0.log" Oct 30 21:11:34.210845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:34.210814 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z2rbr" Oct 30 21:11:35.098045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098003 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:35.098045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098049 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098082 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098102 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098120 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098145 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098164 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098167 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098197 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098223 4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098240 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098262 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098228 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098226 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098280 4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098291 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098296 4921 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098244 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098225128 +0000 UTC m=+17.080925542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098349 4921 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:11:35.098355 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098363 4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098424 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls podName:ddc7b666-f4e0-4a1d-8866-f7cd8b841f99 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098401128 +0000 UTC m=+17.081101534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls") pod "dns-default-dspqr" (UID: "ddc7b666-f4e0-4a1d-8866-f7cd8b841f99") : secret "dns-default-metrics-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098444 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098437918 +0000 UTC m=+17.081138314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : configmap references non-existent config key: service-ca.crt Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098466 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098461337 +0000 UTC m=+17.081161734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098476 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs podName:bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098471317 +0000 UTC m=+17.081171715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs") pod "router-default-6dc5bbb4f7-hmkgf" (UID: "bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a") : secret "router-metrics-certs-default" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098485 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098480269 +0000 UTC m=+17.081180669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098496 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert podName:d478b307-b6de-4b2a-b6cf-3a3b7bbbb414 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098489898 +0000 UTC m=+17.081190294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6bt4k" (UID: "d478b307-b6de-4b2a-b6cf-3a3b7bbbb414") : secret "networking-console-plugin-cert" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098506 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls podName:85c676ae-424f-431f-b349-27eeb74c1c1a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098500207 +0000 UTC m=+17.081200605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-76zdg" (UID: "85c676ae-424f-431f-b349-27eeb74c1c1a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098515 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls podName:26cead8f-43b1-4117-a777-2a571d56a16a nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098509862 +0000 UTC m=+17.081210259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2thm8" (UID: "26cead8f-43b1-4117-a777-2a571d56a16a") : secret "samples-operator-tls" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:35.098530 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098587 4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:11:35.099032 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:35.098628 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs podName:5d316c29-1798-4c5a-9c1a-b8036fc6ccef nodeName:}" failed. No retries permitted until 2026-10-30 21:11:43.098622538 +0000 UTC m=+17.081322958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs") pod "network-metrics-daemon-xrkn6" (UID: "5d316c29-1798-4c5a-9c1a-b8036fc6ccef") : secret "metrics-daemon-secret" not found Oct 30 21:11:36.786251 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:36.786217 4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-220.us-west-1.compute.internal" event="NodeReady" Oct 30 21:11:40.161214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.161180 4921 scope.go:122] "RemoveContainer" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" Oct 30 21:11:40.264412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.264386 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/2.log" Oct 30 21:11:40.264593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.264484 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerStarted","Data":"484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d"} Oct 30 21:11:40.264836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.264813 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:40.266561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.266194 4921 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rvc5s container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Oct 30 21:11:40.266561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:40.266257 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Oct 30 21:11:41.103641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.103599 4921 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rvc5s container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.8:8443/healthz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Oct 30 21:11:41.103901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.103653 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/healthz\": dial tcp 10.132.0.8:8443: connect: connection refused" Oct 30 21:11:41.269386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.269360 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:11:41.269813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.269769 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/2.log" Oct 30 21:11:41.269813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.269801 4921 generic.go:356] "Generic (PLEG): container finished" podID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerID="484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d" exitCode=255 Oct 30 21:11:41.269917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.269831 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerDied","Data":"484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d"} Oct 30 21:11:41.269917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.269905 4921 scope.go:122] "RemoveContainer" containerID="e0a2a73804883b353c277b4109ee9698da70561c57db1c70950642dd4ec9eccc" Oct 30 21:11:41.270163 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:41.270140 4921 scope.go:122] "RemoveContainer" containerID="484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d" Oct 30 21:11:41.270392 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:41.270368 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:42.274297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:42.274270 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:11:42.274759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:42.274637 4921 scope.go:122] "RemoveContainer" containerID="484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d" Oct 30 21:11:42.274840 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:42.274820 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:42.337287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:42.337250 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t"] Oct 30 21:11:42.337477 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:42.337451 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-799568db4-pqw6t" podUID="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" Oct 30 21:11:43.177156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177115 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177168 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") pod \"image-registry-799568db4-pqw6t\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177217 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") pod \"image-registry-b9ddc8f6f-shs2w\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177249 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177262 4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177277 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177291 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177331 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-799568db4-pqw6t: secret "image-registry-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177340 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177355 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls podName:7c440558-5078-4b4d-b28a-2c36109ab98e nodeName:}" failed. No retries permitted until 2026-10-30 21:11:59.17733466 +0000 UTC m=+33.160035072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-sjz89" (UID: "7c440558-5078-4b4d-b28a-2c36109ab98e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177365 4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177385 4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b9ddc8f6f-shs2w: secret "image-registry-tls" not found Oct 30 21:11:43.177386 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177393 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls podName:3f9eb4e9-d614-40d7-a31f-e8fd8cc30893 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:59.177375722 +0000 UTC m=+33.160076143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls") pod "image-registry-799568db4-pqw6t" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893") : secret "image-registry-tls" not found Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:43.177426 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls podName:d3304c86-a470-4066-ace5-5927eb040db4 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:59.177412614 +0000 UTC m=+33.160113025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls") pod "image-registry-b9ddc8f6f-shs2w" (UID: "d3304c86-a470-4066-ace5-5927eb040db4") : secret "image-registry-tls" not found Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177421 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177480 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177519 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177547 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:43.178036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.177694 4921 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:11:43.178353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.178102 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-service-ca-bundle\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:43.179972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.179946 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d316c29-1798-4c5a-9c1a-b8036fc6ccef-metrics-certs\") pod \"network-metrics-daemon-xrkn6\" (UID: \"5d316c29-1798-4c5a-9c1a-b8036fc6ccef\") " pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:43.180095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.180052 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a-metrics-certs\") pod \"router-default-6dc5bbb4f7-hmkgf\" (UID: \"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a\") " pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:43.180394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.180368 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85c676ae-424f-431f-b349-27eeb74c1c1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-76zdg\" (UID: \"85c676ae-424f-431f-b349-27eeb74c1c1a\") " pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:43.180483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.180444 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddc7b666-f4e0-4a1d-8866-f7cd8b841f99-metrics-tls\") pod \"dns-default-dspqr\" (UID: \"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99\") " pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:43.180523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.180452 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d478b307-b6de-4b2a-b6cf-3a3b7bbbb414-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6bt4k\" (UID: \"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:43.180588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.180573 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/26cead8f-43b1-4117-a777-2a571d56a16a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2thm8\" (UID: \"26cead8f-43b1-4117-a777-2a571d56a16a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:43.181212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.181200 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrkn6" Oct 30 21:11:43.184583 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.184565 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:43.185670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.185651 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-76zdg" Oct 30 21:11:43.277170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.277140 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:43.284232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.284213 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:43.379290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379253 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379301 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379346 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379411 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379487 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379527 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379564 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") pod \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\" (UID: \"3f9eb4e9-d614-40d7-a31f-e8fd8cc30893\") " Oct 30 21:11:43.379803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379716 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:11:43.379803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379765 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:43.379909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379819 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:43.379909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379883 4921 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-certificates\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.379909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379903 4921 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-trusted-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.380048 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.379922 4921 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-ca-trust-extracted\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.381738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.381712 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:43.381856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.381759 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:11:43.381856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.381829 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "kube-api-access-cxw2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:43.381856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.381837 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration" pod "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" (UID: "3f9eb4e9-d614-40d7-a31f-e8fd8cc30893"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:11:43.475829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.475782 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:43.477303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.477281 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" Oct 30 21:11:43.477401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.477379 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" Oct 30 21:11:43.481175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.481151 4921 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-bound-sa-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.481272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.481177 4921 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-image-registry-private-configuration\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.481272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.481190 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw2t\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-kube-api-access-cxw2t\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.481272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.481200 4921 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-installation-pull-secrets\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:43.617370 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.617343 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrkn6"] Oct 30 21:11:43.620645 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:43.620611 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d316c29_1798_4c5a_9c1a_b8036fc6ccef.slice/crio-dfe957cb193a7b8089c9e57fd4562e8a83b9bdfa134bc0a63731724bc917185a WatchSource:0}: Error finding container dfe957cb193a7b8089c9e57fd4562e8a83b9bdfa134bc0a63731724bc917185a: Status 404 returned error can't find the container with id dfe957cb193a7b8089c9e57fd4562e8a83b9bdfa134bc0a63731724bc917185a Oct 30 21:11:43.620740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.620656 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-76zdg"] Oct 30 21:11:43.621222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.621184 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dspqr"] Oct 30 21:11:43.623514 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:43.623492 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c676ae_424f_431f_b349_27eeb74c1c1a.slice/crio-9b4c04ce2096d728ae65a044a06d84a683b7762b64591f6e1f0a6e2525a2189e WatchSource:0}: Error finding container 9b4c04ce2096d728ae65a044a06d84a683b7762b64591f6e1f0a6e2525a2189e: Status 404 returned error can't find the container with id 9b4c04ce2096d728ae65a044a06d84a683b7762b64591f6e1f0a6e2525a2189e Oct 30 21:11:43.660966 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:43.660942 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc7b666_f4e0_4a1d_8866_f7cd8b841f99.slice/crio-a68624a279d393fc643dfb64a1b608cd32d29e3e3e01175d8935922a977b1603 WatchSource:0}: Error finding container a68624a279d393fc643dfb64a1b608cd32d29e3e3e01175d8935922a977b1603: Status 404 returned error can't find the container with id a68624a279d393fc643dfb64a1b608cd32d29e3e3e01175d8935922a977b1603 Oct 30 21:11:43.854989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.854901 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6bt4k"] Oct 30 21:11:43.858369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.858343 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8"] Oct 30 21:11:43.858624 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:43.858594 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd478b307_b6de_4b2a_b6cf_3a3b7bbbb414.slice/crio-676213e2a49c7a2bc0196d2f263bfe6c8ba5443b234043e804b43acd3cc625c7 WatchSource:0}: Error finding container 676213e2a49c7a2bc0196d2f263bfe6c8ba5443b234043e804b43acd3cc625c7: Status 404 returned error can't find the container with id 676213e2a49c7a2bc0196d2f263bfe6c8ba5443b234043e804b43acd3cc625c7 Oct 30 21:11:43.862197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:43.862178 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6dc5bbb4f7-hmkgf"] Oct 30 21:11:43.865090 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:43.865066 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc208c6c_8b3d_4eb0_b7e6_ab2daef5578a.slice/crio-f3f045ba2e998f5ce295a24e765d686ac06bf3f6a7357afd7de93f1422567131 WatchSource:0}: Error finding container f3f045ba2e998f5ce295a24e765d686ac06bf3f6a7357afd7de93f1422567131: Status 404 returned error can't find the container with id f3f045ba2e998f5ce295a24e765d686ac06bf3f6a7357afd7de93f1422567131 Oct 30 21:11:44.282111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.282073 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dspqr" event={"ID":"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99","Type":"ContainerStarted","Data":"a68624a279d393fc643dfb64a1b608cd32d29e3e3e01175d8935922a977b1603"} Oct 30 21:11:44.283224 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.283201 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" event={"ID":"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414","Type":"ContainerStarted","Data":"676213e2a49c7a2bc0196d2f263bfe6c8ba5443b234043e804b43acd3cc625c7"} Oct 30 21:11:44.284431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.284409 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrkn6" event={"ID":"5d316c29-1798-4c5a-9c1a-b8036fc6ccef","Type":"ContainerStarted","Data":"dfe957cb193a7b8089c9e57fd4562e8a83b9bdfa134bc0a63731724bc917185a"} Oct 30 21:11:44.285955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.285937 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" event={"ID":"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a","Type":"ContainerStarted","Data":"cc38a3b7d7d4bb04221177d048c3d002980f8c3fb380bf6d5965253bb9dd48be"} Oct 30 21:11:44.286008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.285968 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" event={"ID":"bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a","Type":"ContainerStarted","Data":"f3f045ba2e998f5ce295a24e765d686ac06bf3f6a7357afd7de93f1422567131"} Oct 30 21:11:44.287604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.287584 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-76zdg" event={"ID":"85c676ae-424f-431f-b349-27eeb74c1c1a","Type":"ContainerStarted","Data":"fe1d221148bd20257f38ee3067b73a9530afe67b2dcd16d94695058435d7ad6c"} Oct 30 21:11:44.287712 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.287631 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-76zdg" event={"ID":"85c676ae-424f-431f-b349-27eeb74c1c1a","Type":"ContainerStarted","Data":"9b4c04ce2096d728ae65a044a06d84a683b7762b64591f6e1f0a6e2525a2189e"} Oct 30 21:11:44.288738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.288724 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799568db4-pqw6t" Oct 30 21:11:44.288793 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.288748 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" event={"ID":"26cead8f-43b1-4117-a777-2a571d56a16a","Type":"ContainerStarted","Data":"4b2e33a2390bbc297a772ac94d35e7067d2e805cba87e7e05b6e958975d7fbec"} Oct 30 21:11:44.476847 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.476809 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:44.479759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.479732 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:44.576250 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.576215 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t"] Oct 30 21:11:44.581951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.581920 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799568db4-pqw6t"] Oct 30 21:11:44.690410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:44.690349 4921 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893-registry-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:45.310738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:45.310634 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrkn6" event={"ID":"5d316c29-1798-4c5a-9c1a-b8036fc6ccef","Type":"ContainerStarted","Data":"486f4caa791f10fd621dde530b6cd90df67ede478bff2df5775c924f4934938b"} Oct 30 21:11:45.314246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:45.313865 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-76zdg" event={"ID":"85c676ae-424f-431f-b349-27eeb74c1c1a","Type":"ContainerStarted","Data":"f20fa3ff1b6641632ae0a86ef57593a5d3780d169bdb06b853da1ccc5bd324ed"} Oct 30 21:11:45.314246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:45.313932 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:45.315997 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:45.315805 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6dc5bbb4f7-hmkgf" Oct 30 21:11:45.501423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:45.501379 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9ddc8f6f-shs2w"] Oct 30 21:11:45.502168 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:45.502025 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" podUID="d3304c86-a470-4066-ace5-5927eb040db4" Oct 30 21:11:46.169359 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.169272 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9eb4e9-d614-40d7-a31f-e8fd8cc30893" path="/var/lib/kubelet/pods/3f9eb4e9-d614-40d7-a31f-e8fd8cc30893/volumes" Oct 30 21:11:46.318925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.318499 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6bt4k" event={"ID":"d478b307-b6de-4b2a-b6cf-3a3b7bbbb414","Type":"ContainerStarted","Data":"9cb087ae53c7d66410a7526937d44baea7281f68c1429c07240a10531bf7dfc1"} Oct 30 21:11:46.321040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.320712 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:46.321687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.321152 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrkn6" event={"ID":"5d316c29-1798-4c5a-9c1a-b8036fc6ccef","Type":"ContainerStarted","Data":"b570b8c94e3ec848d7ad3f76cd900f8eef011627faeb1922b0e8afebdedd3ce9"} Oct 30 21:11:46.327866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.327760 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410496 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410559 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410606 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410655 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410694 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410753 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.410846 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") pod \"d3304c86-a470-4066-ace5-5927eb040db4\" (UID: \"d3304c86-a470-4066-ace5-5927eb040db4\") " Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.412305 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:46.412912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.412702 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:11:46.413615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.413578 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:11:46.417819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.417793 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "kube-api-access-msqkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:46.418067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.418037 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:11:46.418999 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.418961 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:11:46.420759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.420726 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token" pod "d3304c86-a470-4066-ace5-5927eb040db4" (UID: "d3304c86-a470-4066-ace5-5927eb040db4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:46.512183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512150 4921 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-registry-certificates\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512183 4921 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-image-registry-private-configuration\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512197 4921 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3304c86-a470-4066-ace5-5927eb040db4-ca-trust-extracted\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512207 4921 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-bound-sa-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512216 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqkl\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-kube-api-access-msqkl\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512225 4921 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3304c86-a470-4066-ace5-5927eb040db4-installation-pull-secrets\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:46.512434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:46.512233 4921 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3304c86-a470-4066-ace5-5927eb040db4-trusted-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:47.325765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.325727 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dspqr" event={"ID":"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99","Type":"ContainerStarted","Data":"c7ee18f9a64da72555ee07d48d977f3ebd22729fe425924b475950c50b52da97"} Oct 30 21:11:47.326229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.325784 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dspqr" event={"ID":"ddc7b666-f4e0-4a1d-8866-f7cd8b841f99","Type":"ContainerStarted","Data":"971aa5fdd1e376f7e6d0dd43d67c6955c10db9d9ff7e53a704ff832c5d00c0cc"} Oct 30 21:11:47.326229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.325830 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:47.327911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.327886 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-76zdg" event={"ID":"85c676ae-424f-431f-b349-27eeb74c1c1a","Type":"ContainerStarted","Data":"f20841f889ee57fb495d53d7aaccfc6a03e2700d63710d6a3fd954efa18fa0e8"} Oct 30 21:11:47.329776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.329756 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" event={"ID":"26cead8f-43b1-4117-a777-2a571d56a16a","Type":"ContainerStarted","Data":"ad9d497172af0a8b50de04499f7b3cf2b7079de055f4feef1187db4f511ff5b7"} Oct 30 21:11:47.329880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.329796 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2thm8" event={"ID":"26cead8f-43b1-4117-a777-2a571d56a16a","Type":"ContainerStarted","Data":"ceccd29fe20e6093d8accf52297090749f7be1b49812500f36273a87ec98ea0d"} Oct 30 21:11:47.329880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.329857 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b9ddc8f6f-shs2w" Oct 30 21:11:47.617650 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.617568 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b9ddc8f6f-shs2w"] Oct 30 21:11:47.621147 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.621064 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b9ddc8f6f-shs2w"] Oct 30 21:11:47.722696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:47.722660 4921 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3304c86-a470-4066-ace5-5927eb040db4-registry-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:11:48.165453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:48.165420 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3304c86-a470-4066-ace5-5927eb040db4" path="/var/lib/kubelet/pods/d3304c86-a470-4066-ace5-5927eb040db4/volumes" Oct 30 21:11:51.102823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:51.102792 4921 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:11:51.103268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:51.103158 4921 scope.go:122] "RemoveContainer" containerID="484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d" Oct 30 21:11:51.103382 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:11:51.103362 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rvc5s_openshift-console-operator(4b295f49-70ab-4d12-b08c-57f3dfb5324b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" Oct 30 21:11:56.715654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:56.715626 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6d887" Oct 30 21:11:57.338235 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:57.338198 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dspqr" Oct 30 21:11:58.495442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:58.495411 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k67dh" Oct 30 21:11:59.216723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:59.216684 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:59.219192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:59.219166 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c440558-5078-4b4d-b28a-2c36109ab98e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-sjz89\" (UID: \"7c440558-5078-4b4d-b28a-2c36109ab98e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:59.376723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:59.376676 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" Oct 30 21:11:59.784087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:11:59.784016 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89"] Oct 30 21:11:59.787456 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:11:59.787412 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c440558_5078_4b4d_b28a_2c36109ab98e.slice/crio-b6b96bdb3b3e4978f66188224af199ab5d397d457bb9b3d9407aa604c4c5f894 WatchSource:0}: Error finding container b6b96bdb3b3e4978f66188224af199ab5d397d457bb9b3d9407aa604c4c5f894: Status 404 returned error can't find the container with id b6b96bdb3b3e4978f66188224af199ab5d397d457bb9b3d9407aa604c4c5f894 Oct 30 21:12:00.376655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:00.376609 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" event={"ID":"7c440558-5078-4b4d-b28a-2c36109ab98e","Type":"ContainerStarted","Data":"b6b96bdb3b3e4978f66188224af199ab5d397d457bb9b3d9407aa604c4c5f894"} Oct 30 21:12:02.161055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.161017 4921 scope.go:122] "RemoveContainer" containerID="484d712c9bdd52c320583d4d88778f3d8921d692d575681b18562e60a919f07d" Oct 30 21:12:02.386860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.386833 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:12:02.387051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.386951 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" event={"ID":"4b295f49-70ab-4d12-b08c-57f3dfb5324b","Type":"ContainerStarted","Data":"4feb520a4918c080c575bfe9b4ec701f478f731f09a0eb48e68891f78e87a82f"} Oct 30 21:12:02.387941 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.387922 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:12:02.388457 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.388433 4921 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rvc5s container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Oct 30 21:12:02.388512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.388482 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" podUID="4b295f49-70ab-4d12-b08c-57f3dfb5324b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Oct 30 21:12:02.390587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:02.390559 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-sjz89" event={"ID":"7c440558-5078-4b4d-b28a-2c36109ab98e","Type":"ContainerStarted","Data":"6a0e09a9a0f99dec292be8b7bc75c6dd25ab24f987c4664e1f633fe37304b16b"} Oct 30 21:12:03.398253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:03.398222 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rvc5s" Oct 30 21:12:07.539884 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:12:07.539800 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:12:13.975011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:13.974971 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x7d46"] Oct 30 21:12:13.977245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:13.977228 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.078182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.078140 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:12:14.078383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.078249 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:12:14.108835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.108804 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:12:14.127645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.127613 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wxcc9\"" Oct 30 21:12:14.127645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.127613 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:12:14.128898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.128877 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.128954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.128914 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-tls\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.128954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.128941 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-wtmp\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129054 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129087 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7dv\" (UniqueName: \"kubernetes.io/projected/33465666-e6b5-42fe-838a-e8ea65b65870-kube-api-access-jc7dv\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129147 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-root\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129188 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-sys\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129219 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-textfile\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.129276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.129237 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-metrics-client-ca\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230033 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-tls\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230070 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-wtmp\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230090 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230112 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7dv\" (UniqueName: \"kubernetes.io/projected/33465666-e6b5-42fe-838a-e8ea65b65870-kube-api-access-jc7dv\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230134 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-root\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230163 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-sys\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230189 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-textfile\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230209 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-metrics-client-ca\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230237 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230249 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-wtmp\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230250 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-root\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230298 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33465666-e6b5-42fe-838a-e8ea65b65870-sys\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230615 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-textfile\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230759 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-accelerators-collector-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.230936 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.230914 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33465666-e6b5-42fe-838a-e8ea65b65870-metrics-client-ca\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.232502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.232474 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.232609 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.232561 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/33465666-e6b5-42fe-838a-e8ea65b65870-node-exporter-tls\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.285596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.285570 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7dv\" (UniqueName: \"kubernetes.io/projected/33465666-e6b5-42fe-838a-e8ea65b65870-kube-api-access-jc7dv\") pod \"node-exporter-x7d46\" (UID: \"33465666-e6b5-42fe-838a-e8ea65b65870\") " pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.289491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.289463 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x7d46" Oct 30 21:12:14.300157 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:12:14.300135 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33465666_e6b5_42fe_838a_e8ea65b65870.slice/crio-26f0bfe1983d6b0d3d38c6ecc7516e1230adf5b82a9003d8441f517a146f8697 WatchSource:0}: Error finding container 26f0bfe1983d6b0d3d38c6ecc7516e1230adf5b82a9003d8441f517a146f8697: Status 404 returned error can't find the container with id 26f0bfe1983d6b0d3d38c6ecc7516e1230adf5b82a9003d8441f517a146f8697 Oct 30 21:12:14.432520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:14.432484 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7d46" event={"ID":"33465666-e6b5-42fe-838a-e8ea65b65870","Type":"ContainerStarted","Data":"26f0bfe1983d6b0d3d38c6ecc7516e1230adf5b82a9003d8441f517a146f8697"} Oct 30 21:12:16.444826 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:16.444787 4921 generic.go:356] "Generic (PLEG): container finished" podID="33465666-e6b5-42fe-838a-e8ea65b65870" containerID="01cd56fd7ea4146d81b24d2b78f91e8b61e4e5f1849f1a0ee307ba990d9087ab" exitCode=0 Oct 30 21:12:16.445209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:16.444873 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7d46" event={"ID":"33465666-e6b5-42fe-838a-e8ea65b65870","Type":"ContainerDied","Data":"01cd56fd7ea4146d81b24d2b78f91e8b61e4e5f1849f1a0ee307ba990d9087ab"} Oct 30 21:12:17.450373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:17.450334 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7d46" event={"ID":"33465666-e6b5-42fe-838a-e8ea65b65870","Type":"ContainerStarted","Data":"0b7ad9e48a5e25c791449ae791233aabaac2adfd28a1151d7ff66e4043ce1e82"} Oct 30 21:12:17.450867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:17.450402 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x7d46" event={"ID":"33465666-e6b5-42fe-838a-e8ea65b65870","Type":"ContainerStarted","Data":"c3c0fd953651fe2d7f39b9cb632d4982ed1c52d6bf317dcca5e381f9dff47a68"} Oct 30 21:12:17.643927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:17.643866 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x7d46" podStartSLOduration=3.257408844 podStartE2EDuration="4.643850509s" totalImagesPullingTime="1.319581137s" totalInitContainerRuntime="66.860528ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:17.615088178 +0000 UTC m=+51.597788596" watchObservedRunningTime="2026-10-30 21:12:17.643850509 +0000 UTC m=+51.626550943" Oct 30 21:12:31.501471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:31.501440 4921 generic.go:356] "Generic (PLEG): container finished" podID="53efca0c-3152-434c-998e-d0916b982591" containerID="33c91b02c42a80bcb81ca94edcb69ecb3de1c8c967f2a54e07375501182dd905" exitCode=0 Oct 30 21:12:31.501822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:31.501502 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" event={"ID":"53efca0c-3152-434c-998e-d0916b982591","Type":"ContainerDied","Data":"33c91b02c42a80bcb81ca94edcb69ecb3de1c8c967f2a54e07375501182dd905"} Oct 30 21:12:31.501873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:31.501822 4921 scope.go:122] "RemoveContainer" containerID="33c91b02c42a80bcb81ca94edcb69ecb3de1c8c967f2a54e07375501182dd905" Oct 30 21:12:32.507120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:32.507085 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96rhm" event={"ID":"53efca0c-3152-434c-998e-d0916b982591","Type":"ContainerStarted","Data":"0953e451cdcf0b1a5ddde63aad514a14da705e75a4eb8a74b532a8c06fa6d7c0"} Oct 30 21:12:37.540457 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:12:37.540418 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:12:42.544252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:42.544217 4921 generic.go:356] "Generic (PLEG): container finished" podID="89bb38ba-2a52-40fc-9b88-a21200c1c542" containerID="09da65e64f182dd55efcf9a9b734c0abf641dcfa0b361dbc57e7b400c59e2a3a" exitCode=0 Oct 30 21:12:42.544702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:42.544275 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" event={"ID":"89bb38ba-2a52-40fc-9b88-a21200c1c542","Type":"ContainerDied","Data":"09da65e64f182dd55efcf9a9b734c0abf641dcfa0b361dbc57e7b400c59e2a3a"} Oct 30 21:12:42.544702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:42.544605 4921 scope.go:122] "RemoveContainer" containerID="09da65e64f182dd55efcf9a9b734c0abf641dcfa0b361dbc57e7b400c59e2a3a" Oct 30 21:12:43.549305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:43.549269 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjg6j" event={"ID":"89bb38ba-2a52-40fc-9b88-a21200c1c542","Type":"ContainerStarted","Data":"8250d31494c15174e7ab9b7cc7f23d046184c40bd2072d3f2cf04926d00e5f76"} Oct 30 21:12:56.598774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:56.598737 4921 generic.go:356] "Generic (PLEG): container finished" podID="5f0ca892-6436-4c1b-994c-bb97ba977307" containerID="f9d88ef1c437244ed299faf480d67e7ae34960ad13a72d6b7d3646b1712a0a17" exitCode=0 Oct 30 21:12:56.599157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:56.598804 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs27v" event={"ID":"5f0ca892-6436-4c1b-994c-bb97ba977307","Type":"ContainerDied","Data":"f9d88ef1c437244ed299faf480d67e7ae34960ad13a72d6b7d3646b1712a0a17"} Oct 30 21:12:56.599157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:56.599108 4921 scope.go:122] "RemoveContainer" containerID="f9d88ef1c437244ed299faf480d67e7ae34960ad13a72d6b7d3646b1712a0a17" Oct 30 21:12:57.604288 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:12:57.604255 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-rs27v" event={"ID":"5f0ca892-6436-4c1b-994c-bb97ba977307","Type":"ContainerStarted","Data":"34d419fa006ecd1a19f3bab783b7781ffea182b88afee735b7e205d1bec9b7e4"} Oct 30 21:13:07.539169 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:13:07.539128 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:13:37.540231 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:13:37.540142 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:14:07.539033 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:14:07.538993 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:14:37.538651 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:14:37.538610 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:15:07.538300 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:15:07.538224 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:15:37.538629 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:15:37.538592 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:16:07.538952 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:16:07.538915 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:16:22.898945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:22.898869 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr"] Oct 30 21:16:22.902236 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:22.902218 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:22.993049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:22.993013 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:22.993049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:22.993047 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:22.993266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:22.993068 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfbt\" (UniqueName: \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.005188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.005153 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:16:23.005401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.005282 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:16:23.005401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.005328 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-x8t6s\"" Oct 30 21:16:23.006368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.006344 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr"] Oct 30 21:16:23.094236 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.094199 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.094236 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.094233 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.094532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.094255 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfbt\" (UniqueName: \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.094724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.094696 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.094797 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.094705 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.151173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.151100 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfbt\" (UniqueName: \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.220072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.220029 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:16:23.589555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:23.589523 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr"] Oct 30 21:16:23.592900 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:23.592869 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd69a72c_8eee_4d7a_8fed_b2dc7d63c231.slice/crio-da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c WatchSource:0}: Error finding container da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c: Status 404 returned error can't find the container with id da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c Oct 30 21:16:24.298921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.298885 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerStarted","Data":"da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c"} Oct 30 21:16:24.322347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.322306 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm"] Oct 30 21:16:24.326784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.326769 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.383219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.383186 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm"] Oct 30 21:16:24.416430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.416397 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd"] Oct 30 21:16:24.419650 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.419632 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.420480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.420459 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2m9ln\"" Oct 30 21:16:24.494618 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.494581 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd"] Oct 30 21:16:24.506682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.506641 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.506868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.506696 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.506868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.506803 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfzk\" (UniqueName: \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.520722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.520688 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k"] Oct 30 21:16:24.524160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.524132 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-s7m9v\"" Oct 30 21:16:24.524652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.524634 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.575950 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.575870 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-x7zc5\"" Oct 30 21:16:24.583220 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.583184 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k"] Oct 30 21:16:24.607346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607293 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.607551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607358 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.607551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607391 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtbn\" (UniqueName: \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.607551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607420 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfzk\" (UniqueName: \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.607551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607449 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.607551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607523 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.608002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.607972 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.608193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.608169 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.613571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.613543 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck"] Oct 30 21:16:24.617915 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.617892 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.664727 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.664693 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfzk\" (UniqueName: \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:24.676465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.674911 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-5pvfr\"" Oct 30 21:16:24.676465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.676397 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck"] Oct 30 21:16:24.708448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708412 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.708660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708471 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9q5\" (UniqueName: \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.708660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708579 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtbn\" (UniqueName: \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.708660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708624 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfj6\" (UniqueName: \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.708660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708656 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.708875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708706 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.708875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708775 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.708875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708842 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.709006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.708937 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.709145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.709125 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.709301 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.709281 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.765177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.765149 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtbn\" (UniqueName: \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:24.809917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.809881 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9q5\" (UniqueName: \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.810088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.809945 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfj6\" (UniqueName: \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.810088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.809971 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.810088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810004 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.810088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810061 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.810333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810110 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.810577 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810547 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.810577 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810558 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.810725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810550 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.810825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.810802 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.867126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.867055 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfj6\" (UniqueName: \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.867743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.867716 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9q5\" (UniqueName: \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:24.935998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.935960 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:16:24.942250 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:24.942217 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:16:25.036674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.036642 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:16:25.143851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.143763 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:16:25.332039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.332004 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm"] Oct 30 21:16:25.361999 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:25.361969 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebaf8f6_4939_4b1a_ada6_588ff35e700c.slice/crio-a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d WatchSource:0}: Error finding container a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d: Status 404 returned error can't find the container with id a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d Oct 30 21:16:25.370597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.370575 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck"] Oct 30 21:16:25.377938 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:25.377911 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3424d812_fceb_48f8_a371_a8c629ae79ae.slice/crio-bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6 WatchSource:0}: Error finding container bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6: Status 404 returned error can't find the container with id bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6 Oct 30 21:16:25.421358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.421294 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd"] Oct 30 21:16:25.423724 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:25.423691 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f64084_a52b_413b_98b8_60d8d229f6b3.slice/crio-a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2 WatchSource:0}: Error finding container a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2: Status 404 returned error can't find the container with id a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2 Oct 30 21:16:25.524421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.524390 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k"] Oct 30 21:16:25.528221 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:25.528188 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39473baf_db93_41ba_a1cc_734a89501156.slice/crio-b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf WatchSource:0}: Error finding container b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf: Status 404 returned error can't find the container with id b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf Oct 30 21:16:25.933463 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.933427 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4"] Oct 30 21:16:25.938542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.938509 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:25.990340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:25.990285 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-swv8l\"" Oct 30 21:16:26.040793 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.040755 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4"] Oct 30 21:16:26.122602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.122570 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.122602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.122608 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.122818 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.122637 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zhp\" (UniqueName: \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.169833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.169806 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:16:26.170139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.170119 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:16:26.224036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.224000 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.224036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.224034 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.224299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.224072 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zhp\" (UniqueName: \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.224475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.224452 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.224531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.224513 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.281257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.281222 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zhp\" (UniqueName: \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.310711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.310665 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" event={"ID":"62f64084-a52b-413b-98b8-60d8d229f6b3","Type":"ContainerStarted","Data":"a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2"} Oct 30 21:16:26.312621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.312530 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" event={"ID":"0ebaf8f6-4939-4b1a-ada6-588ff35e700c","Type":"ContainerStarted","Data":"a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d"} Oct 30 21:16:26.314400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.314352 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" event={"ID":"3424d812-fceb-48f8-a371-a8c629ae79ae","Type":"ContainerStarted","Data":"bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6"} Oct 30 21:16:26.316946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.316920 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerStarted","Data":"b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf"} Oct 30 21:16:26.608451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.608354 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-swv8l\"" Oct 30 21:16:26.618459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.618164 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:16:26.830308 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.830271 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn"] Oct 30 21:16:26.835586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.835559 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:26.892244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.890254 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wjcsq\"" Oct 30 21:16:26.892244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.892130 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn"] Oct 30 21:16:26.933785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.933729 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:26.933971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.933927 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:26.934036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:26.934009 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275bv\" (UniqueName: \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.034631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.034478 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.034631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.034549 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.034905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.034867 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275bv\" (UniqueName: \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.034974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.034906 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.035416 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.035394 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.066056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.065983 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4"] Oct 30 21:16:27.072815 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:27.072781 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39148fde_49f6_4864_825f_a21f3f1e34e9.slice/crio-193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19 WatchSource:0}: Error finding container 193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19: Status 404 returned error can't find the container with id 193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19 Oct 30 21:16:27.095778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.095741 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275bv\" (UniqueName: \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.157576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.157489 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:16:27.324348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.324236 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" event={"ID":"39148fde-49f6-4864-825f-a21f3f1e34e9","Type":"ContainerStarted","Data":"193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19"} Oct 30 21:16:27.562446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:27.562409 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn"] Oct 30 21:16:27.567493 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:27.567458 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccf1026_9da8_4344_a45a_3d1baa9b1624.slice/crio-35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0 WatchSource:0}: Error finding container 35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0: Status 404 returned error can't find the container with id 35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0 Oct 30 21:16:28.334272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:28.334233 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerStarted","Data":"35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0"} Oct 30 21:16:29.545139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.543242 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx"] Oct 30 21:16:29.547658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.547629 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.618366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.612555 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx"] Oct 30 21:16:29.659461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.659172 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffst\" (UniqueName: \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.659461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.659269 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.659953 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.659879 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.760447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.760402 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffst\" (UniqueName: \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.760627 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.760510 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.760674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.760639 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.761113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.761087 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.761807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.761782 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.820408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.820333 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffst\" (UniqueName: \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:29.871034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:29.870607 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:16:30.319865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.319786 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx"] Oct 30 21:16:30.329960 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:30.329922 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0cde62_0e92_40ac_b685_332baaac5e33.slice/crio-d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a WatchSource:0}: Error finding container d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a: Status 404 returned error can't find the container with id d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a Oct 30 21:16:30.335744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.335720 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:16:30.369285 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.369200 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerStarted","Data":"d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a"} Oct 30 21:16:30.522865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.522825 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:16:30.528391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.528308 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.580932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.580844 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:16:30.581538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.580979 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6bm8h\"" Oct 30 21:16:30.633241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.633204 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:16:30.671334 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671288 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671345 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxs99\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671382 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671400 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671427 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671442 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671476 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671491 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.671837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.671541 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.772948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.772907 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.772962 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxs99\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773020 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773049 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773086 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773108 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773164 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773188 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.773501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773236 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.775688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773668 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.775688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.773848 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.775688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.774335 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.775688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.774980 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.775688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.775646 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.786582 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.786470 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.789670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.789590 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.834966 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.834840 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd"] Oct 30 21:16:30.839004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.838981 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.858937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.858902 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.859864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.859806 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxs99\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\") pod \"maas-default-gateway-openshift-default-6cc458d887-ggsrf\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:30.891478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.891432 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-2s45h\"" Oct 30 21:16:30.905632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.905540 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd"] Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.975832 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdzg\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-kube-api-access-kjdzg\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.975888 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeeb498-3174-46e9-a86c-ade577195b67-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.975919 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.975989 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.976017 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.976069 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.976102 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.976131 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:30.976413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:30.976164 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeeb498-3174-46e9-a86c-ade577195b67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.077575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077532 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdzg\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-kube-api-access-kjdzg\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.077762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077588 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeeb498-3174-46e9-a86c-ade577195b67-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.077762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077621 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.077762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077698 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.077762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077723 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.078007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077775 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.078007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077815 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.078007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077845 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.078007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.077873 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeeb498-3174-46e9-a86c-ade577195b67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.078796 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.078765 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aeeb498-3174-46e9-a86c-ade577195b67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.080535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.079893 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.080535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.079981 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.080535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.080237 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.080535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.080443 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.082864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.082838 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aeeb498-3174-46e9-a86c-ade577195b67-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.092876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.092810 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aeeb498-3174-46e9-a86c-ade577195b67-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.142267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.142231 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdzg\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-kube-api-access-kjdzg\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.143403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.143374 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aeeb498-3174-46e9-a86c-ade577195b67-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd\" (UID: \"3aeeb498-3174-46e9-a86c-ade577195b67\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.158963 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.158915 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:31.159174 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.158931 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:31.622161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.622050 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:16:31.630390 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:31.630353 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af6193a_70ed_409d_9fed_7f7e41d590cb.slice/crio-d3fc506a6a4b6004b529e7f94f28a042e397287996473c714767c2b7d3a0b430 WatchSource:0}: Error finding container d3fc506a6a4b6004b529e7f94f28a042e397287996473c714767c2b7d3a0b430: Status 404 returned error can't find the container with id d3fc506a6a4b6004b529e7f94f28a042e397287996473c714767c2b7d3a0b430 Oct 30 21:16:31.673011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.672249 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd"] Oct 30 21:16:31.782466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.782429 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg"] Oct 30 21:16:31.788306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.788277 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.834266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.834227 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs"] Oct 30 21:16:31.838851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.838778 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:31.844444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.844375 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg"] Oct 30 21:16:31.858061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.858025 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w"] Oct 30 21:16:31.862836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.862806 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:31.885717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.885682 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs"] Oct 30 21:16:31.894223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.894136 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.894438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.894261 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.895350 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.895308 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkjf\" (UniqueName: \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.902162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.902111 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl"] Oct 30 21:16:31.906607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.905831 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:31.918651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.918620 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w"] Oct 30 21:16:31.979060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.978987 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl"] Oct 30 21:16:31.997528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997486 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkjf\" (UniqueName: \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.997695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997557 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpkp\" (UniqueName: \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:31.997695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997592 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:31.997695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997635 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:31.997695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997670 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:31.997823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997697 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:31.997823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997747 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.997823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997773 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhn9t\" (UniqueName: \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:31.997920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997898 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.997953 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997937 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:31.997987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997975 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2679p\" (UniqueName: \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:31.998023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.997995 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:31.998476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.998455 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:31.999345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:31.999205 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:32.054675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.054642 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkjf\" (UniqueName: \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.098774 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.098841 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.098878 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.098905 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.098957 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhn9t\" (UniqueName: \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.099008 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.099040 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2679p\" (UniqueName: \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.099072 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.099143 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpkp\" (UniqueName: \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.099902 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.100232 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.100552 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.100790 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.101301 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.102067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.101761 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.112730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.112447 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:16:32.164638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.163469 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2679p\" (UniqueName: \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.164638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.164414 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:16:32.167707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.167673 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpkp\" (UniqueName: \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.177357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.174610 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhn9t\" (UniqueName: \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.199918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.199624 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:16:32.235177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.233387 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:16:32.403759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.403591 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" event={"ID":"3aeeb498-3174-46e9-a86c-ade577195b67","Type":"ContainerStarted","Data":"f1fb20b567757ec320bfc9e648eb4ecca6b2b2f5c5f2ddbadbaaeea42a1a6bde"} Oct 30 21:16:32.410989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.410880 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerStarted","Data":"d3fc506a6a4b6004b529e7f94f28a042e397287996473c714767c2b7d3a0b430"} Oct 30 21:16:32.560846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.560733 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg"] Oct 30 21:16:32.567787 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:32.567747 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67cf244_777d_449f_876c_75ab23861ecf.slice/crio-70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2 WatchSource:0}: Error finding container 70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2: Status 404 returned error can't find the container with id 70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2 Oct 30 21:16:32.629978 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.629922 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w"] Oct 30 21:16:32.639014 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:32.638944 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eeca8d8_82d0_412f_95ce_40a57c423e32.slice/crio-22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411 WatchSource:0}: Error finding container 22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411: Status 404 returned error can't find the container with id 22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411 Oct 30 21:16:32.681726 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.681687 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs"] Oct 30 21:16:32.687506 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:32.687415 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0726b154_b664_4b59_ac1b_cdbdca88de91.slice/crio-1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037 WatchSource:0}: Error finding container 1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037: Status 404 returned error can't find the container with id 1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037 Oct 30 21:16:32.716288 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:32.716253 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl"] Oct 30 21:16:32.725330 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:16:32.725276 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e53672_e510_4eb3_81eb_063fd61df23b.slice/crio-61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087 WatchSource:0}: Error finding container 61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087: Status 404 returned error can't find the container with id 61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087 Oct 30 21:16:33.423593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:33.423523 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerStarted","Data":"61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087"} Oct 30 21:16:33.426777 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:33.426740 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" event={"ID":"a67cf244-777d-449f-876c-75ab23861ecf","Type":"ContainerStarted","Data":"70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2"} Oct 30 21:16:33.428988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:33.428912 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerStarted","Data":"22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411"} Oct 30 21:16:33.443085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:33.443041 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" event={"ID":"0726b154-b664-4b59-ac1b-cdbdca88de91","Type":"ContainerStarted","Data":"1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037"} Oct 30 21:16:37.638712 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:16:37.638510 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:16:41.548517 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:41.548477 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerStarted","Data":"d869a02d54b163b5f3efc7d1549fdda08cafb0c968eaa880b3bce4f0abad42fd"} Oct 30 21:16:42.563382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:42.563340 4921 generic.go:356] "Generic (PLEG): container finished" podID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerID="d869a02d54b163b5f3efc7d1549fdda08cafb0c968eaa880b3bce4f0abad42fd" exitCode=0 Oct 30 21:16:42.563890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:42.563467 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerDied","Data":"d869a02d54b163b5f3efc7d1549fdda08cafb0c968eaa880b3bce4f0abad42fd"} Oct 30 21:16:45.833860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.833816 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:45.834289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.833913 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:45.834289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.833956 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:45.858179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.857652 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:45.858179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.857742 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:45.858179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:45.857789 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:16:46.618813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.618773 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerStarted","Data":"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005"} Oct 30 21:16:46.624238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.624192 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" event={"ID":"3aeeb498-3174-46e9-a86c-ade577195b67","Type":"ContainerStarted","Data":"8bb0edb020db7012a4efd8ae7bb7d100dcf7829fea14e06766f62b693c5f874f"} Oct 30 21:16:46.629301 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.629261 4921 generic.go:356] "Generic (PLEG): container finished" podID="db0cde62-0e92-40ac-b685-332baaac5e33" containerID="fe71cccec3a7f203d8898e4c2c01c08edb239d34a39ce76425eb7d9293996464" exitCode=0 Oct 30 21:16:46.629519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.629367 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerDied","Data":"fe71cccec3a7f203d8898e4c2c01c08edb239d34a39ce76425eb7d9293996464"} Oct 30 21:16:46.788197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.786157 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podStartSLOduration=2.587044485 podStartE2EDuration="16.786136802s" totalImagesPullingTime="14.199092317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:46.785188222 +0000 UTC m=+320.767888644" watchObservedRunningTime="2026-10-30 21:16:46.786136802 +0000 UTC m=+320.768837222" Oct 30 21:16:46.907429 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:46.907301 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" podStartSLOduration=2.734266947 podStartE2EDuration="16.9072817s" totalImagesPullingTime="14.173014753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:46.905946442 +0000 UTC m=+320.888646864" watchObservedRunningTime="2026-10-30 21:16:46.9072817 +0000 UTC m=+320.889982120" Oct 30 21:16:47.165934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.159587 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:47.165934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.159657 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:47.172730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.172186 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:47.172730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.172621 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:47.642351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.641916 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:47.642351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.642010 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:47.642351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.642186 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd" Oct 30 21:16:47.642742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:47.642719 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:16:48.648810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.648772 4921 generic.go:356] "Generic (PLEG): container finished" podID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerID="64d3510ecbd730d47d33f6fc0dc6c520d626380e856c835de306cee3bfd08344" exitCode=0 Oct 30 21:16:48.649303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.648875 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" event={"ID":"62f64084-a52b-413b-98b8-60d8d229f6b3","Type":"ContainerDied","Data":"64d3510ecbd730d47d33f6fc0dc6c520d626380e856c835de306cee3bfd08344"} Oct 30 21:16:48.651741 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.651424 4921 generic.go:356] "Generic (PLEG): container finished" podID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerID="37130e6b409787ba0aa54f4cacd6b426eb90b034c9ec36af510c426f9c118e7e" exitCode=0 Oct 30 21:16:48.651741 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.651541 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerDied","Data":"37130e6b409787ba0aa54f4cacd6b426eb90b034c9ec36af510c426f9c118e7e"} Oct 30 21:16:48.655601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.654769 4921 generic.go:356] "Generic (PLEG): container finished" podID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerID="87618037de3dd134611c61b7c4c4eeee564980f614daefb9d53acc4f332a2e95" exitCode=0 Oct 30 21:16:48.655601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.654857 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" event={"ID":"0ebaf8f6-4939-4b1a-ada6-588ff35e700c","Type":"ContainerDied","Data":"87618037de3dd134611c61b7c4c4eeee564980f614daefb9d53acc4f332a2e95"} Oct 30 21:16:48.659111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.659081 4921 generic.go:356] "Generic (PLEG): container finished" podID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerID="7347feb121393f7f1af6f6566e00f4c9eed5621ef4ce773b17dcf6064fb20813" exitCode=0 Oct 30 21:16:48.659418 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.659385 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" event={"ID":"3424d812-fceb-48f8-a371-a8c629ae79ae","Type":"ContainerDied","Data":"7347feb121393f7f1af6f6566e00f4c9eed5621ef4ce773b17dcf6064fb20813"} Oct 30 21:16:48.662761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.662731 4921 generic.go:356] "Generic (PLEG): container finished" podID="39473baf-db93-41ba-a1cc-734a89501156" containerID="35da5c285dee3925e4d03b5eb070458552c031594890b7f19dfe6801b76043b4" exitCode=0 Oct 30 21:16:48.663015 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.662882 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerDied","Data":"35da5c285dee3925e4d03b5eb070458552c031594890b7f19dfe6801b76043b4"} Oct 30 21:16:48.666973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.666943 4921 generic.go:356] "Generic (PLEG): container finished" podID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerID="fee156b9fc571411f73ec3da312d3a62ed36f33f8b769715a409b333f283467d" exitCode=0 Oct 30 21:16:48.667116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.667097 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" event={"ID":"0726b154-b664-4b59-ac1b-cdbdca88de91","Type":"ContainerDied","Data":"fee156b9fc571411f73ec3da312d3a62ed36f33f8b769715a409b333f283467d"} Oct 30 21:16:48.672426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.672385 4921 generic.go:356] "Generic (PLEG): container finished" podID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerID="1b0c36c7a7226b986de09fd1ef8ea5da49163dff0d257291c9260a3873f7e24d" exitCode=0 Oct 30 21:16:48.672663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.672523 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerDied","Data":"1b0c36c7a7226b986de09fd1ef8ea5da49163dff0d257291c9260a3873f7e24d"} Oct 30 21:16:48.674965 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.674924 4921 generic.go:356] "Generic (PLEG): container finished" podID="a67cf244-777d-449f-876c-75ab23861ecf" containerID="aa7a5b49f609a7acf289dd0e19e55aacbccadd0f910a4e4e7da345420deb6a1b" exitCode=0 Oct 30 21:16:48.675367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:48.674997 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" event={"ID":"a67cf244-777d-449f-876c-75ab23861ecf","Type":"ContainerDied","Data":"aa7a5b49f609a7acf289dd0e19e55aacbccadd0f910a4e4e7da345420deb6a1b"} Oct 30 21:16:49.682499 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.682461 4921 generic.go:356] "Generic (PLEG): container finished" podID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerID="62415ed5caaa8fc66dc8667d8576350974488ef4e24b1f4fd13e2ebf658c21cf" exitCode=0 Oct 30 21:16:49.682964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.682559 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerDied","Data":"62415ed5caaa8fc66dc8667d8576350974488ef4e24b1f4fd13e2ebf658c21cf"} Oct 30 21:16:49.685018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.684980 4921 generic.go:356] "Generic (PLEG): container finished" podID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerID="523682ac8e101f40d41da3a5531f5bae9befac2f2a08510a763c778d8ea93f4c" exitCode=0 Oct 30 21:16:49.685121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.685065 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" event={"ID":"39148fde-49f6-4864-825f-a21f3f1e34e9","Type":"ContainerDied","Data":"523682ac8e101f40d41da3a5531f5bae9befac2f2a08510a763c778d8ea93f4c"} Oct 30 21:16:49.686731 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.686705 4921 generic.go:356] "Generic (PLEG): container finished" podID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerID="29c5a6b890ca2db750bf22df16ac481367b8e951168de850578e8d79cbb6a79e" exitCode=0 Oct 30 21:16:49.686870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:49.686801 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerDied","Data":"29c5a6b890ca2db750bf22df16ac481367b8e951168de850578e8d79cbb6a79e"} Oct 30 21:16:50.705572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.705197 4921 generic.go:356] "Generic (PLEG): container finished" podID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerID="d7d6286cc75fd3bbfa23413be6943ba6cb6bee758fd80a1a916971e5659d58dd" exitCode=0 Oct 30 21:16:50.705572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.705438 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerDied","Data":"d7d6286cc75fd3bbfa23413be6943ba6cb6bee758fd80a1a916971e5659d58dd"} Oct 30 21:16:50.712711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.710457 4921 generic.go:356] "Generic (PLEG): container finished" podID="a67cf244-777d-449f-876c-75ab23861ecf" containerID="8b9f55553cb35a842b5b098b46391dec1ef6113b760e360adb707fb84aca6583" exitCode=0 Oct 30 21:16:50.712711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.710578 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" event={"ID":"a67cf244-777d-449f-876c-75ab23861ecf","Type":"ContainerDied","Data":"8b9f55553cb35a842b5b098b46391dec1ef6113b760e360adb707fb84aca6583"} Oct 30 21:16:50.715591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.715543 4921 generic.go:356] "Generic (PLEG): container finished" podID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerID="434acc80042d06b86e5dedcecf381c99b4be160f9b019edcb239ecfe0f0dc0b9" exitCode=0 Oct 30 21:16:50.715695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.715670 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" event={"ID":"62f64084-a52b-413b-98b8-60d8d229f6b3","Type":"ContainerDied","Data":"434acc80042d06b86e5dedcecf381c99b4be160f9b019edcb239ecfe0f0dc0b9"} Oct 30 21:16:50.718453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.718415 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerStarted","Data":"33d55771ccdf51c6fb9768b8dff07b30dc13fe6d097832e78ed172c9fce8b28c"} Oct 30 21:16:50.721816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.721791 4921 generic.go:356] "Generic (PLEG): container finished" podID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerID="3dab8d43532d8f8c191e33fb977ad651f8cd0d0403ea1c07d40c2dcf347a93c1" exitCode=0 Oct 30 21:16:50.721914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.721868 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" event={"ID":"0ebaf8f6-4939-4b1a-ada6-588ff35e700c","Type":"ContainerDied","Data":"3dab8d43532d8f8c191e33fb977ad651f8cd0d0403ea1c07d40c2dcf347a93c1"} Oct 30 21:16:50.734902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.734834 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerStarted","Data":"67b8c7b281ee09264c6c2acdbdabe9246eb740ee03e556f962946f02dcbf1190"} Oct 30 21:16:50.737879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.737659 4921 generic.go:356] "Generic (PLEG): container finished" podID="db0cde62-0e92-40ac-b685-332baaac5e33" containerID="be5dbd7f5f0b2dd8b1431111bb92d4c1bbf77c122b9d3a794143763b7fd53c77" exitCode=0 Oct 30 21:16:50.737879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:50.737751 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerDied","Data":"be5dbd7f5f0b2dd8b1431111bb92d4c1bbf77c122b9d3a794143763b7fd53c77"} Oct 30 21:16:51.803227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.802248 4921 generic.go:356] "Generic (PLEG): container finished" podID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerID="33d55771ccdf51c6fb9768b8dff07b30dc13fe6d097832e78ed172c9fce8b28c" exitCode=0 Oct 30 21:16:51.803227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.802391 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerDied","Data":"33d55771ccdf51c6fb9768b8dff07b30dc13fe6d097832e78ed172c9fce8b28c"} Oct 30 21:16:51.815876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.815833 4921 generic.go:356] "Generic (PLEG): container finished" podID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerID="1b88419d513b796adb405776d357068944ce05c087e7455cce999cd50b7e2ec0" exitCode=0 Oct 30 21:16:51.816051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.815955 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" event={"ID":"3424d812-fceb-48f8-a371-a8c629ae79ae","Type":"ContainerDied","Data":"1b88419d513b796adb405776d357068944ce05c087e7455cce999cd50b7e2ec0"} Oct 30 21:16:51.828155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.827356 4921 generic.go:356] "Generic (PLEG): container finished" podID="39473baf-db93-41ba-a1cc-734a89501156" containerID="67b8c7b281ee09264c6c2acdbdabe9246eb740ee03e556f962946f02dcbf1190" exitCode=0 Oct 30 21:16:51.828155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.827646 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerDied","Data":"67b8c7b281ee09264c6c2acdbdabe9246eb740ee03e556f962946f02dcbf1190"} Oct 30 21:16:51.838493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.837425 4921 generic.go:356] "Generic (PLEG): container finished" podID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerID="62a60917f470cb4a451e13ca6a3459ff9f516121d5737f7ab375eba4b909bac7" exitCode=0 Oct 30 21:16:51.838493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.837534 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerDied","Data":"62a60917f470cb4a451e13ca6a3459ff9f516121d5737f7ab375eba4b909bac7"} Oct 30 21:16:51.843120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.842966 4921 generic.go:356] "Generic (PLEG): container finished" podID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerID="1458349deb49e2d88ddbcb3ffb6df7a159f5d9146a8973c30cd5bbeb878b11d2" exitCode=0 Oct 30 21:16:51.843120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:16:51.843020 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" event={"ID":"0726b154-b664-4b59-ac1b-cdbdca88de91","Type":"ContainerDied","Data":"1458349deb49e2d88ddbcb3ffb6df7a159f5d9146a8973c30cd5bbeb878b11d2"} Oct 30 21:17:00.016753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:00.016705 4921 generic.go:356] "Generic (PLEG): container finished" podID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerID="890bca899602bcd1b19f410a48b9277f7b3dc677d349498cd7f367396122cd2f" exitCode=0 Oct 30 21:17:00.017294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:00.016767 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" event={"ID":"39148fde-49f6-4864-825f-a21f3f1e34e9","Type":"ContainerDied","Data":"890bca899602bcd1b19f410a48b9277f7b3dc677d349498cd7f367396122cd2f"} Oct 30 21:17:07.557388 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:17:07.557347 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:17:13.180277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:13.180190 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerStarted","Data":"06a0d555216964d72847057c3a53537575ef3fee550e4d3ea24f373198d07a6a"} Oct 30 21:17:13.298068 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:13.297997 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" podStartSLOduration=3.698088993 podStartE2EDuration="51.297977261s" totalImagesPullingTime="47.599888268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:13.296503839 +0000 UTC m=+347.279204258" watchObservedRunningTime="2026-10-30 21:17:13.297977261 +0000 UTC m=+347.280677681" Oct 30 21:17:14.223062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:14.222625 4921 generic.go:356] "Generic (PLEG): container finished" podID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerID="06a0d555216964d72847057c3a53537575ef3fee550e4d3ea24f373198d07a6a" exitCode=0 Oct 30 21:17:14.223062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:14.222698 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerDied","Data":"06a0d555216964d72847057c3a53537575ef3fee550e4d3ea24f373198d07a6a"} Oct 30 21:17:15.518428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.518071 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:17:15.641705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.641666 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\") pod \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " Oct 30 21:17:15.641852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.641802 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\" (UniqueName: \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\") pod \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " Oct 30 21:17:15.641917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.641891 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\") pod \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\" (UID: \"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231\") " Oct 30 21:17:15.643244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.643158 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle" pod "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" (UID: "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:15.645762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.645731 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt" pod "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" (UID: "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231"). InnerVolumeSpecName "kube-api-access-bwfbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:15.655516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.655084 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util" pod "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" (UID: "fd69a72c-8eee-4d7a-8fed-b2dc7d63c231"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:15.748984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.748945 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:15.749111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.748991 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:15.749111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:15.749007 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwfbt\" (UniqueName: \"kubernetes.io/projected/fd69a72c-8eee-4d7a-8fed-b2dc7d63c231-kube-api-access-bwfbt\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:16.250865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.250830 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerStarted","Data":"ce16b79511f1af5e6dbd697350853213fd84b02014775db71f429a0ea3ec3791"} Oct 30 21:17:16.256727 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.256688 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" event={"ID":"fd69a72c-8eee-4d7a-8fed-b2dc7d63c231","Type":"ContainerDied","Data":"da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c"} Oct 30 21:17:16.256888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.256742 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da73218792d9aa13dfebb0e042f984f929b1637fc930c9e0e68e2cc92f0ace1c" Oct 30 21:17:16.256888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.256812 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwllzr" Oct 30 21:17:16.263061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.263020 4921 generic.go:356] "Generic (PLEG): container finished" podID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerID="05a9cc441238c5182e7555df6f0dba32b3c1adb4da59a9ef0e3f5e1ec0af8927" exitCode=0 Oct 30 21:17:16.263221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.263062 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" event={"ID":"39148fde-49f6-4864-825f-a21f3f1e34e9","Type":"ContainerDied","Data":"05a9cc441238c5182e7555df6f0dba32b3c1adb4da59a9ef0e3f5e1ec0af8927"} Oct 30 21:17:16.269481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.269447 4921 generic.go:356] "Generic (PLEG): container finished" podID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerID="e8386404e80cc6c376dc0528c186c62b1f925ef6d4eb6f21326cb64d1d1a8bff" exitCode=0 Oct 30 21:17:16.269653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.269511 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" event={"ID":"0726b154-b664-4b59-ac1b-cdbdca88de91","Type":"ContainerDied","Data":"e8386404e80cc6c376dc0528c186c62b1f925ef6d4eb6f21326cb64d1d1a8bff"} Oct 30 21:17:16.370587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:16.369764 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" podStartSLOduration=2.814706104 podStartE2EDuration="45.36974476s" totalImagesPullingTime="42.555038656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:16.367070211 +0000 UTC m=+350.349770632" watchObservedRunningTime="2026-10-30 21:17:16.36974476 +0000 UTC m=+350.352445181" Oct 30 21:17:17.279343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.278518 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerStarted","Data":"8df634b77638ad3a8b63b628f73aec98ae31dd5c570f50834e1467f767d3de5e"} Oct 30 21:17:17.282695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.282069 4921 generic.go:356] "Generic (PLEG): container finished" podID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerID="f547dc13d648073b56a2015f3d36f13665ee215a7a403017d8cc6c34909176ae" exitCode=0 Oct 30 21:17:17.282695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.282173 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" event={"ID":"0ebaf8f6-4939-4b1a-ada6-588ff35e700c","Type":"ContainerDied","Data":"f547dc13d648073b56a2015f3d36f13665ee215a7a403017d8cc6c34909176ae"} Oct 30 21:17:17.291209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.291002 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerStarted","Data":"9ef2fa4176cb2fdd4c0d5ad9a39425e43722a9fc7552812db439d1ecbc935695"} Oct 30 21:17:17.294529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.294507 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerStarted","Data":"62e1cc42f36d0638489ddbd47036046bb4fcacb83dab010fbf0dd67e980da588"} Oct 30 21:17:17.297816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.297784 4921 generic.go:356] "Generic (PLEG): container finished" podID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerID="ce16b79511f1af5e6dbd697350853213fd84b02014775db71f429a0ea3ec3791" exitCode=0 Oct 30 21:17:17.297915 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.297868 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerDied","Data":"ce16b79511f1af5e6dbd697350853213fd84b02014775db71f429a0ea3ec3791"} Oct 30 21:17:17.300795 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.300665 4921 generic.go:356] "Generic (PLEG): container finished" podID="a67cf244-777d-449f-876c-75ab23861ecf" containerID="dc332cfbd627e1d452d3ebaedd55d92a75ae08c0fb7df276309ad7972a7dc254" exitCode=0 Oct 30 21:17:17.300795 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.300742 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" event={"ID":"a67cf244-777d-449f-876c-75ab23861ecf","Type":"ContainerDied","Data":"dc332cfbd627e1d452d3ebaedd55d92a75ae08c0fb7df276309ad7972a7dc254"} Oct 30 21:17:17.396966 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.396902 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" podStartSLOduration=3.7223441409999998 podStartE2EDuration="46.396885951s" totalImagesPullingTime="42.67454181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:17.394878894 +0000 UTC m=+351.377579314" watchObservedRunningTime="2026-10-30 21:17:17.396885951 +0000 UTC m=+351.379586372" Oct 30 21:17:17.452228 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.452198 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:17:17.499670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.499256 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:17:17.567165 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.567130 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\" (UniqueName: \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\") pod \"0726b154-b664-4b59-ac1b-cdbdca88de91\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " Oct 30 21:17:17.567346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.567219 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\") pod \"0726b154-b664-4b59-ac1b-cdbdca88de91\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " Oct 30 21:17:17.567346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.567249 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\") pod \"0726b154-b664-4b59-ac1b-cdbdca88de91\" (UID: \"0726b154-b664-4b59-ac1b-cdbdca88de91\") " Oct 30 21:17:17.567723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.567689 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle" pod "0726b154-b664-4b59-ac1b-cdbdca88de91" (UID: "0726b154-b664-4b59-ac1b-cdbdca88de91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:17.570183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.570152 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p" pod "0726b154-b664-4b59-ac1b-cdbdca88de91" (UID: "0726b154-b664-4b59-ac1b-cdbdca88de91"). InnerVolumeSpecName "kube-api-access-2679p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:17.575459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.575429 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util" pod "0726b154-b664-4b59-ac1b-cdbdca88de91" (UID: "0726b154-b664-4b59-ac1b-cdbdca88de91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:17.668443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668392 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\") pod \"39148fde-49f6-4864-825f-a21f3f1e34e9\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " Oct 30 21:17:17.668628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668510 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\") pod \"39148fde-49f6-4864-825f-a21f3f1e34e9\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " Oct 30 21:17:17.668706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668635 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\" (UniqueName: \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\") pod \"39148fde-49f6-4864-825f-a21f3f1e34e9\" (UID: \"39148fde-49f6-4864-825f-a21f3f1e34e9\") " Oct 30 21:17:17.668881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668859 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2679p\" (UniqueName: \"kubernetes.io/projected/0726b154-b664-4b59-ac1b-cdbdca88de91-kube-api-access-2679p\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.669004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668885 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.669004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668901 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0726b154-b664-4b59-ac1b-cdbdca88de91-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.669004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.668874 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle" pod "39148fde-49f6-4864-825f-a21f3f1e34e9" (UID: "39148fde-49f6-4864-825f-a21f3f1e34e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:17.671104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.671066 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp" pod "39148fde-49f6-4864-825f-a21f3f1e34e9" (UID: "39148fde-49f6-4864-825f-a21f3f1e34e9"). InnerVolumeSpecName "kube-api-access-47zhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:17.675600 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.675565 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util" pod "39148fde-49f6-4864-825f-a21f3f1e34e9" (UID: "39148fde-49f6-4864-825f-a21f3f1e34e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:17.741678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.741624 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" podStartSLOduration=3.749439831 podStartE2EDuration="48.741610808s" totalImagesPullingTime="44.992170977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:17.739821592 +0000 UTC m=+351.722522024" watchObservedRunningTime="2026-10-30 21:17:17.741610808 +0000 UTC m=+351.724311223" Oct 30 21:17:17.770056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.770018 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.770056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.770056 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zhp\" (UniqueName: \"kubernetes.io/projected/39148fde-49f6-4864-825f-a21f3f1e34e9-kube-api-access-47zhp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.770258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.770073 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39148fde-49f6-4864-825f-a21f3f1e34e9-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.971087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:17.971026 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" podStartSLOduration=3.894904913 podStartE2EDuration="51.971006676s" totalImagesPullingTime="48.076101763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:17.970236954 +0000 UTC m=+351.952937373" watchObservedRunningTime="2026-10-30 21:17:17.971006676 +0000 UTC m=+351.953707096" Oct 30 21:17:18.307814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.307732 4921 generic.go:356] "Generic (PLEG): container finished" podID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerID="384b06ad19a0587d8e5a5f780c844a2eaacb16ee2d3a0b98b1086ed1367609ac" exitCode=0 Oct 30 21:17:18.307814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.307803 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" event={"ID":"62f64084-a52b-413b-98b8-60d8d229f6b3","Type":"ContainerDied","Data":"384b06ad19a0587d8e5a5f780c844a2eaacb16ee2d3a0b98b1086ed1367609ac"} Oct 30 21:17:18.309971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.309945 4921 generic.go:356] "Generic (PLEG): container finished" podID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerID="8df634b77638ad3a8b63b628f73aec98ae31dd5c570f50834e1467f767d3de5e" exitCode=0 Oct 30 21:17:18.310100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.310021 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerDied","Data":"8df634b77638ad3a8b63b628f73aec98ae31dd5c570f50834e1467f767d3de5e"} Oct 30 21:17:18.312015 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.311995 4921 generic.go:356] "Generic (PLEG): container finished" podID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerID="6b26ff02aa44e1c852c15f6fdde262f2a7a37738523fc551f66396f42c3713da" exitCode=0 Oct 30 21:17:18.312113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.312067 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" event={"ID":"3424d812-fceb-48f8-a371-a8c629ae79ae","Type":"ContainerDied","Data":"6b26ff02aa44e1c852c15f6fdde262f2a7a37738523fc551f66396f42c3713da"} Oct 30 21:17:18.313559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.313543 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:17:18.314153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.314133 4921 generic.go:356] "Generic (PLEG): container finished" podID="39473baf-db93-41ba-a1cc-734a89501156" containerID="34d2d96a2b5cc97258cd374fcac210172720d02ef537de5bcd0c0116cab826ed" exitCode=1 Oct 30 21:17:18.314255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.314209 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerDied","Data":"34d2d96a2b5cc97258cd374fcac210172720d02ef537de5bcd0c0116cab826ed"} Oct 30 21:17:18.315855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.315836 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" event={"ID":"39148fde-49f6-4864-825f-a21f3f1e34e9","Type":"ContainerDied","Data":"193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19"} Oct 30 21:17:18.315855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.315846 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kwmz4" Oct 30 21:17:18.316007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.315866 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193594398a1afbce46fd5e9b8117489af1f36279641f6be4f8135b3fd70c1e19" Oct 30 21:17:18.318072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.318046 4921 generic.go:356] "Generic (PLEG): container finished" podID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerID="9ef2fa4176cb2fdd4c0d5ad9a39425e43722a9fc7552812db439d1ecbc935695" exitCode=0 Oct 30 21:17:18.318182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.318102 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerDied","Data":"9ef2fa4176cb2fdd4c0d5ad9a39425e43722a9fc7552812db439d1ecbc935695"} Oct 30 21:17:18.320208 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.320191 4921 generic.go:356] "Generic (PLEG): container finished" podID="db0cde62-0e92-40ac-b685-332baaac5e33" containerID="62e1cc42f36d0638489ddbd47036046bb4fcacb83dab010fbf0dd67e980da588" exitCode=0 Oct 30 21:17:18.320295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.320248 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerDied","Data":"62e1cc42f36d0638489ddbd47036046bb4fcacb83dab010fbf0dd67e980da588"} Oct 30 21:17:18.322045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.322029 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" Oct 30 21:17:18.322134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.322057 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648pbs" event={"ID":"0726b154-b664-4b59-ac1b-cdbdca88de91","Type":"ContainerDied","Data":"1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037"} Oct 30 21:17:18.322134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.322088 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dba82288751eb9e44df33b6bd139b3ada879727e2d18b532bcac7d469ba4037" Oct 30 21:17:18.459737 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.459699 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:17:18.470233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.470202 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:17:18.514224 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.514202 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:17:18.576496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576410 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\" (UniqueName: \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\") pod \"a67cf244-777d-449f-876c-75ab23861ecf\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " Oct 30 21:17:18.576496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576454 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\" (UniqueName: \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\") pod \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " Oct 30 21:17:18.576496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576491 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\") pod \"a67cf244-777d-449f-876c-75ab23861ecf\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " Oct 30 21:17:18.576732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576590 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\") pod \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " Oct 30 21:17:18.576732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576676 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\") pod \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\" (UID: \"0ebaf8f6-4939-4b1a-ada6-588ff35e700c\") " Oct 30 21:17:18.576835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.576743 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\") pod \"a67cf244-777d-449f-876c-75ab23861ecf\" (UID: \"a67cf244-777d-449f-876c-75ab23861ecf\") " Oct 30 21:17:18.577261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.577229 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle" pod "0ebaf8f6-4939-4b1a-ada6-588ff35e700c" (UID: "0ebaf8f6-4939-4b1a-ada6-588ff35e700c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.577435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.577402 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle" pod "a67cf244-777d-449f-876c-75ab23861ecf" (UID: "a67cf244-777d-449f-876c-75ab23861ecf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.578728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.578704 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf" pod "a67cf244-777d-449f-876c-75ab23861ecf" (UID: "a67cf244-777d-449f-876c-75ab23861ecf"). InnerVolumeSpecName "kube-api-access-5kkjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:18.578728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.578725 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk" pod "0ebaf8f6-4939-4b1a-ada6-588ff35e700c" (UID: "0ebaf8f6-4939-4b1a-ada6-588ff35e700c"). InnerVolumeSpecName "kube-api-access-5qfzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:18.583038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.583015 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util" pod "a67cf244-777d-449f-876c-75ab23861ecf" (UID: "a67cf244-777d-449f-876c-75ab23861ecf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.585029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.584996 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util" pod "0ebaf8f6-4939-4b1a-ada6-588ff35e700c" (UID: "0ebaf8f6-4939-4b1a-ada6-588ff35e700c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.677747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.677710 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\") pod \"c8e53672-e510-4eb3-81eb-063fd61df23b\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " Oct 30 21:17:18.677944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.677849 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\") pod \"c8e53672-e510-4eb3-81eb-063fd61df23b\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " Oct 30 21:17:18.677944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.677901 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\" (UniqueName: \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\") pod \"c8e53672-e510-4eb3-81eb-063fd61df23b\" (UID: \"c8e53672-e510-4eb3-81eb-063fd61df23b\") " Oct 30 21:17:18.678135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678119 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678141 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkjf\" (UniqueName: \"kubernetes.io/projected/a67cf244-777d-449f-876c-75ab23861ecf-kube-api-access-5kkjf\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678156 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qfzk\" (UniqueName: \"kubernetes.io/projected/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-kube-api-access-5qfzk\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678173 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67cf244-777d-449f-876c-75ab23861ecf-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678188 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678201 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ebaf8f6-4939-4b1a-ada6-588ff35e700c-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.678392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.678264 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle" pod "c8e53672-e510-4eb3-81eb-063fd61df23b" (UID: "c8e53672-e510-4eb3-81eb-063fd61df23b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.679967 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.679942 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp" pod "c8e53672-e510-4eb3-81eb-063fd61df23b" (UID: "c8e53672-e510-4eb3-81eb-063fd61df23b"). InnerVolumeSpecName "kube-api-access-xcpkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:18.682754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.682734 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util" pod "c8e53672-e510-4eb3-81eb-063fd61df23b" (UID: "c8e53672-e510-4eb3-81eb-063fd61df23b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:18.779116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.779081 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpkp\" (UniqueName: \"kubernetes.io/projected/c8e53672-e510-4eb3-81eb-063fd61df23b-kube-api-access-xcpkp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.779116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.779112 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:18.779116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:18.779125 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8e53672-e510-4eb3-81eb-063fd61df23b-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.329296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.329259 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" Oct 30 21:17:19.329871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.329264 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbfpzm" event={"ID":"0ebaf8f6-4939-4b1a-ada6-588ff35e700c","Type":"ContainerDied","Data":"a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d"} Oct 30 21:17:19.329871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.329479 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07eb0f0d600dcb6546def8e7a7967463f6611f52ea4486e11de6e6cfec53d8d" Oct 30 21:17:19.332741 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.332704 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" event={"ID":"c8e53672-e510-4eb3-81eb-063fd61df23b","Type":"ContainerDied","Data":"61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087"} Oct 30 21:17:19.332881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.332786 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b055a865f512bc934eae8551b2cf3640f91b23fc447b92ef38dd1aed07b087" Oct 30 21:17:19.332881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.332861 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wf4wl" Oct 30 21:17:19.342838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.342807 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" event={"ID":"a67cf244-777d-449f-876c-75ab23861ecf","Type":"ContainerDied","Data":"70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2"} Oct 30 21:17:19.342970 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.342853 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a5292bfa8f5df62c1ca5dc0780cc3636751c7f1dadd16792051a3ffe27b9a2" Oct 30 21:17:19.343903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.343880 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjw6kg" Oct 30 21:17:19.493152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.493129 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:17:19.581640 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.581569 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:17:19.587813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.587753 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\") pod \"db0cde62-0e92-40ac-b685-332baaac5e33\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " Oct 30 21:17:19.587972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.587818 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\" (UniqueName: \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\") pod \"db0cde62-0e92-40ac-b685-332baaac5e33\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " Oct 30 21:17:19.587972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.587850 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\") pod \"db0cde62-0e92-40ac-b685-332baaac5e33\" (UID: \"db0cde62-0e92-40ac-b685-332baaac5e33\") " Oct 30 21:17:19.588775 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.588731 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle" pod "db0cde62-0e92-40ac-b685-332baaac5e33" (UID: "db0cde62-0e92-40ac-b685-332baaac5e33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.593041 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.592883 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst" pod "db0cde62-0e92-40ac-b685-332baaac5e33" (UID: "db0cde62-0e92-40ac-b685-332baaac5e33"). InnerVolumeSpecName "kube-api-access-nffst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.594513 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.594485 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util" pod "db0cde62-0e92-40ac-b685-332baaac5e33" (UID: "db0cde62-0e92-40ac-b685-332baaac5e33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.605848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.605813 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:17:19.609503 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.608667 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:17:19.614716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.614692 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:17:19.616158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.616136 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:17:19.617976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.617957 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:17:19.688868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.688828 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\") pod \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " Oct 30 21:17:19.689043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.688898 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\" (UniqueName: \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\") pod \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " Oct 30 21:17:19.689043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.688917 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\") pod \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\" (UID: \"9ccf1026-9da8-4344-a45a-3d1baa9b1624\") " Oct 30 21:17:19.689161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.689063 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.689161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.689080 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffst\" (UniqueName: \"kubernetes.io/projected/db0cde62-0e92-40ac-b685-332baaac5e33-kube-api-access-nffst\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.689161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.689094 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db0cde62-0e92-40ac-b685-332baaac5e33-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.689462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.689436 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle" pod "9ccf1026-9da8-4344-a45a-3d1baa9b1624" (UID: "9ccf1026-9da8-4344-a45a-3d1baa9b1624"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.690971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.690939 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv" pod "9ccf1026-9da8-4344-a45a-3d1baa9b1624" (UID: "9ccf1026-9da8-4344-a45a-3d1baa9b1624"). InnerVolumeSpecName "kube-api-access-275bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.694353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.694329 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util" pod "9ccf1026-9da8-4344-a45a-3d1baa9b1624" (UID: "9ccf1026-9da8-4344-a45a-3d1baa9b1624"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.790144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790109 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\" (UniqueName: \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\") pod \"62f64084-a52b-413b-98b8-60d8d229f6b3\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790160 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\") pod \"3424d812-fceb-48f8-a371-a8c629ae79ae\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790193 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\") pod \"3424d812-fceb-48f8-a371-a8c629ae79ae\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790234 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\") pod \"3eeca8d8-82d0-412f-95ce-40a57c423e32\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790254 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\") pod \"39473baf-db93-41ba-a1cc-734a89501156\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790293 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\") pod \"39473baf-db93-41ba-a1cc-734a89501156\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " Oct 30 21:17:19.790365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790352 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\") pod \"62f64084-a52b-413b-98b8-60d8d229f6b3\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " Oct 30 21:17:19.790689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790371 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\" (UniqueName: \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\") pod \"39473baf-db93-41ba-a1cc-734a89501156\" (UID: \"39473baf-db93-41ba-a1cc-734a89501156\") " Oct 30 21:17:19.790689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790408 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\") pod \"62f64084-a52b-413b-98b8-60d8d229f6b3\" (UID: \"62f64084-a52b-413b-98b8-60d8d229f6b3\") " Oct 30 21:17:19.790689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790443 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\") pod \"3eeca8d8-82d0-412f-95ce-40a57c423e32\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " Oct 30 21:17:19.790689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790484 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\" (UniqueName: \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\") pod \"3424d812-fceb-48f8-a371-a8c629ae79ae\" (UID: \"3424d812-fceb-48f8-a371-a8c629ae79ae\") " Oct 30 21:17:19.790689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790530 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\" (UniqueName: \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\") pod \"3eeca8d8-82d0-412f-95ce-40a57c423e32\" (UID: \"3eeca8d8-82d0-412f-95ce-40a57c423e32\") " Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.790987 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle" pod "62f64084-a52b-413b-98b8-60d8d229f6b3" (UID: "62f64084-a52b-413b-98b8-60d8d229f6b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.791682 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle" pod "3eeca8d8-82d0-412f-95ce-40a57c423e32" (UID: "3eeca8d8-82d0-412f-95ce-40a57c423e32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.791864 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle" pod "3424d812-fceb-48f8-a371-a8c629ae79ae" (UID: "3424d812-fceb-48f8-a371-a8c629ae79ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.792222 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle" pod "39473baf-db93-41ba-a1cc-734a89501156" (UID: "39473baf-db93-41ba-a1cc-734a89501156"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.792794 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn" pod "62f64084-a52b-413b-98b8-60d8d229f6b3" (UID: "62f64084-a52b-413b-98b8-60d8d229f6b3"). InnerVolumeSpecName "kube-api-access-pdtbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793427 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5" pod "39473baf-db93-41ba-a1cc-734a89501156" (UID: "39473baf-db93-41ba-a1cc-734a89501156"). InnerVolumeSpecName "kube-api-access-xb9q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793787 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtbn\" (UniqueName: \"kubernetes.io/projected/62f64084-a52b-413b-98b8-60d8d229f6b3-kube-api-access-pdtbn\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793805 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793813 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793823 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.793880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793835 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793953 4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793969 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb9q5\" (UniqueName: \"kubernetes.io/projected/39473baf-db93-41ba-a1cc-734a89501156-kube-api-access-xb9q5\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793983 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275bv\" (UniqueName: \"kubernetes.io/projected/9ccf1026-9da8-4344-a45a-3d1baa9b1624-kube-api-access-275bv\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793898 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6" pod "3424d812-fceb-48f8-a371-a8c629ae79ae" (UID: "3424d812-fceb-48f8-a371-a8c629ae79ae"). InnerVolumeSpecName "kube-api-access-wwfj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.793997 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ccf1026-9da8-4344-a45a-3d1baa9b1624-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.794721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.794261 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t" pod "3eeca8d8-82d0-412f-95ce-40a57c423e32" (UID: "3eeca8d8-82d0-412f-95ce-40a57c423e32"). InnerVolumeSpecName "kube-api-access-qhn9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:19.798646 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.798624 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util" pod "39473baf-db93-41ba-a1cc-734a89501156" (UID: "39473baf-db93-41ba-a1cc-734a89501156"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.799214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.799192 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util" pod "3424d812-fceb-48f8-a371-a8c629ae79ae" (UID: "3424d812-fceb-48f8-a371-a8c629ae79ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.799431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.799407 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util" pod "62f64084-a52b-413b-98b8-60d8d229f6b3" (UID: "62f64084-a52b-413b-98b8-60d8d229f6b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.800259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.800242 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util" pod "3eeca8d8-82d0-412f-95ce-40a57c423e32" (UID: "3eeca8d8-82d0-412f-95ce-40a57c423e32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894718 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62f64084-a52b-413b-98b8-60d8d229f6b3-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894748 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3eeca8d8-82d0-412f-95ce-40a57c423e32-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894761 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfj6\" (UniqueName: \"kubernetes.io/projected/3424d812-fceb-48f8-a371-a8c629ae79ae-kube-api-access-wwfj6\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894772 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhn9t\" (UniqueName: \"kubernetes.io/projected/3eeca8d8-82d0-412f-95ce-40a57c423e32-kube-api-access-qhn9t\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894781 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3424d812-fceb-48f8-a371-a8c629ae79ae-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:19.894804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:19.894789 4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39473baf-db93-41ba-a1cc-734a89501156-util\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:20.349296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.349261 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" event={"ID":"62f64084-a52b-413b-98b8-60d8d229f6b3","Type":"ContainerDied","Data":"a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2"} Oct 30 21:17:20.349778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.349310 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a131f9c08e00cc9cf0806946bd778e572e65370ccc9908cf02d844dda1bcb6a2" Oct 30 21:17:20.349778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.349273 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xh6rd" Oct 30 21:17:20.351538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.351505 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" event={"ID":"3eeca8d8-82d0-412f-95ce-40a57c423e32","Type":"ContainerDied","Data":"22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411"} Oct 30 21:17:20.351641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.351561 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f1437edeef5dbfe5c4b7938f63716fb2e2a1cdeea01117b33952d4cac0d411" Oct 30 21:17:20.351641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.351525 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a47m4w" Oct 30 21:17:20.353539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.353516 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" Oct 30 21:17:20.353720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.353516 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66m6ck" event={"ID":"3424d812-fceb-48f8-a371-a8c629ae79ae","Type":"ContainerDied","Data":"bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6"} Oct 30 21:17:20.353720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.353685 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd31ecc9af13c4d15214bdadb307ec049f67cbb1194a1a57efd66787ad795ff6" Oct 30 21:17:20.354981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.354962 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:17:20.355774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.355755 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" event={"ID":"39473baf-db93-41ba-a1cc-734a89501156","Type":"ContainerDied","Data":"b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf"} Oct 30 21:17:20.355874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.355785 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k" Oct 30 21:17:20.355874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.355825 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b43e0eacfab6d135d0d1cca03fbf26d0733fa25a13fcb614107fb2d929cd2aaf" Oct 30 21:17:20.357898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.357884 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" Oct 30 21:17:20.357985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.357884 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71424jn" event={"ID":"9ccf1026-9da8-4344-a45a-3d1baa9b1624","Type":"ContainerDied","Data":"35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0"} Oct 30 21:17:20.358042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.358004 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35466a4b60e7dbb80118aae3821311180ac10bd1668767bce02d58ea90fe19d0" Oct 30 21:17:20.359970 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.359949 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" event={"ID":"db0cde62-0e92-40ac-b685-332baaac5e33","Type":"ContainerDied","Data":"d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a"} Oct 30 21:17:20.360047 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.359983 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e6bdf88134ca87ddc692f1f642c00b9b98da090c604de7841fa47d8301e98a" Oct 30 21:17:20.360087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:20.360061 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdtvkx" Oct 30 21:17:31.123755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.123717 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw"] Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124043 4921 cpu_manager.go:414] "containerMap: removing container" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124054 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124063 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="extract" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124069 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="extract" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124076 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="util" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124082 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="util" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124094 4921 cpu_manager.go:414] "containerMap: removing container" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124100 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124106 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124111 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124118 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="extract" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124122 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="extract" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124129 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124133 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="pull" Oct 30 21:17:31.124134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124139 4921 cpu_manager.go:414] "containerMap: removing container" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124144 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124151 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124155 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124169 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124176 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124182 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124187 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124195 4921 cpu_manager.go:414] "containerMap: removing container" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124199 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124205 4921 cpu_manager.go:414] "containerMap: removing container" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124209 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124215 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124219 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124226 4921 cpu_manager.go:414] "containerMap: removing container" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124230 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124236 4921 cpu_manager.go:414] "containerMap: removing container" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124241 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124254 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124259 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124269 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124274 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124280 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124284 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124289 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124293 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124300 4921 cpu_manager.go:414] "containerMap: removing container" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124304 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124310 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124327 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124337 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124341 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124351 4921 cpu_manager.go:414] "containerMap: removing container" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124355 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124363 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124367 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124373 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124377 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124383 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124387 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124392 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124396 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124407 4921 cpu_manager.go:414] "containerMap: removing container" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124414 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124420 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124424 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124430 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124433 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="pull" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124439 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124443 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124448 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124452 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124462 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124466 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124473 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124479 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124488 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124493 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="util" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124549 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0726b154-b664-4b59-ac1b-cdbdca88de91" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124557 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="39473baf-db93-41ba-a1cc-734a89501156" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124569 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccf1026-9da8-4344-a45a-3d1baa9b1624" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124574 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebaf8f6-4939-4b1a-ada6-588ff35e700c" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124582 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="39148fde-49f6-4864-825f-a21f3f1e34e9" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124588 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e53672-e510-4eb3-81eb-063fd61df23b" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124594 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67cf244-777d-449f-876c-75ab23861ecf" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124598 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd69a72c-8eee-4d7a-8fed-b2dc7d63c231" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124602 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eeca8d8-82d0-412f-95ce-40a57c423e32" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124606 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3424d812-fceb-48f8-a371-a8c629ae79ae" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124611 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f64084-a52b-413b-98b8-60d8d229f6b3" containerName="extract" Oct 30 21:17:31.124637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.124616 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0cde62-0e92-40ac-b685-332baaac5e33" containerName="extract" Oct 30 21:17:31.127466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.127450 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.171973 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.172008 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-tmp\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.172028 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-manifests\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.172174 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.172204 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.172286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.172225 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn22\" (UniqueName: \"kubernetes.io/projected/c7bbe06a-4326-45cd-949d-a0e511b852eb-kube-api-access-6nn22\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.179657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.179634 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:17:31.179781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.179703 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:17:31.179781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.179726 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:17:31.179781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.179705 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:17:31.228851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.228825 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:17:31.228851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.228825 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-d5dq4\"" Oct 30 21:17:31.246946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.246921 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw"] Oct 30 21:17:31.272786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272758 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.272914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272790 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.272914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272810 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn22\" (UniqueName: \"kubernetes.io/projected/c7bbe06a-4326-45cd-949d-a0e511b852eb-kube-api-access-6nn22\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.272914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272879 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.273072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272913 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-tmp\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.273072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.272946 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-manifests\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.273506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.273455 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-manifests\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.275359 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.275310 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.275458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.275434 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.275501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.275485 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7bbe06a-4326-45cd-949d-a0e511b852eb-tmp\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.275629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.275614 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c7bbe06a-4326-45cd-949d-a0e511b852eb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.329704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.329672 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn22\" (UniqueName: \"kubernetes.io/projected/c7bbe06a-4326-45cd-949d-a0e511b852eb-kube-api-access-6nn22\") pod \"opendatahub-operator-controller-manager-6c776cd74c-bk6kw\" (UID: \"c7bbe06a-4326-45cd-949d-a0e511b852eb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.440062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.439988 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:31.927718 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:31.927692 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw"] Oct 30 21:17:31.929569 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:31.929540 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bbe06a_4326_45cd_949d_a0e511b852eb.slice/crio-4557e9b29d021ca20c82f231cb0fca28d948eb4651cfade81c22a6ba3bf95520 WatchSource:0}: Error finding container 4557e9b29d021ca20c82f231cb0fca28d948eb4651cfade81c22a6ba3bf95520: Status 404 returned error can't find the container with id 4557e9b29d021ca20c82f231cb0fca28d948eb4651cfade81c22a6ba3bf95520 Oct 30 21:17:32.414922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:32.414881 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" event={"ID":"c7bbe06a-4326-45cd-949d-a0e511b852eb","Type":"ContainerStarted","Data":"4557e9b29d021ca20c82f231cb0fca28d948eb4651cfade81c22a6ba3bf95520"} Oct 30 21:17:36.437887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.437793 4921 generic.go:356] "Generic (PLEG): container finished" podID="c7bbe06a-4326-45cd-949d-a0e511b852eb" containerID="aa9739806b21e1393d7d94b4b42f9bf15a2fc463e00b09bf30d9285cfe6e6acc" exitCode=0 Oct 30 21:17:36.438398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.437884 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" event={"ID":"c7bbe06a-4326-45cd-949d-a0e511b852eb","Type":"ContainerDied","Data":"aa9739806b21e1393d7d94b4b42f9bf15a2fc463e00b09bf30d9285cfe6e6acc"} Oct 30 21:17:36.644674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.644644 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj"] Oct 30 21:17:36.646935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.646919 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.720310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.720281 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62m4\" (UniqueName: \"kubernetes.io/projected/09b52524-25c8-423b-abf1-189aa3f0ae11-kube-api-access-s62m4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.720476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.720398 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b52524-25c8-423b-abf1-189aa3f0ae11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.727006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.726989 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:17:36.727006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.726989 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qdmnw\"" Oct 30 21:17:36.727139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.727025 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:17:36.787703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.787678 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj"] Oct 30 21:17:36.821195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.821158 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62m4\" (UniqueName: \"kubernetes.io/projected/09b52524-25c8-423b-abf1-189aa3f0ae11-kube-api-access-s62m4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.821351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.821212 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b52524-25c8-423b-abf1-189aa3f0ae11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.821590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.821574 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b52524-25c8-423b-abf1-189aa3f0ae11-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.879571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.879541 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62m4\" (UniqueName: \"kubernetes.io/projected/09b52524-25c8-423b-abf1-189aa3f0ae11-kube-api-access-s62m4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj\" (UID: \"09b52524-25c8-423b-abf1-189aa3f0ae11\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:36.957673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:36.957637 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" Oct 30 21:17:37.385042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:37.384976 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj"] Oct 30 21:17:37.387502 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:37.387471 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b52524_25c8_423b_abf1_189aa3f0ae11.slice/crio-28b7438fcd9e6db321dee72dd6c93a205533bbf136ca29ae84c3a2944bab5a5d WatchSource:0}: Error finding container 28b7438fcd9e6db321dee72dd6c93a205533bbf136ca29ae84c3a2944bab5a5d: Status 404 returned error can't find the container with id 28b7438fcd9e6db321dee72dd6c93a205533bbf136ca29ae84c3a2944bab5a5d Oct 30 21:17:37.443678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:37.443643 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" event={"ID":"09b52524-25c8-423b-abf1-189aa3f0ae11","Type":"ContainerStarted","Data":"28b7438fcd9e6db321dee72dd6c93a205533bbf136ca29ae84c3a2944bab5a5d"} Oct 30 21:17:37.445742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:37.445713 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" event={"ID":"c7bbe06a-4326-45cd-949d-a0e511b852eb","Type":"ContainerStarted","Data":"38ad91904031b76156e37432622e141c369495711d4b46968f675d1077789be4"} Oct 30 21:17:37.445860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:37.445850 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:37.539040 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:17:37.538953 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:17:37.577692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:37.577644 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" podStartSLOduration=2.674369944 podStartE2EDuration="7.577630307s" totalImagesPullingTime="4.479533247s" totalInitContainerRuntime="423.727116ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:37.576252283 +0000 UTC m=+371.558952701" watchObservedRunningTime="2026-10-30 21:17:37.577630307 +0000 UTC m=+371.560330727" Oct 30 21:17:40.462497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:40.462459 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" event={"ID":"09b52524-25c8-423b-abf1-189aa3f0ae11","Type":"ContainerStarted","Data":"f8f34cf84fd34b5602e607a8f50795a23e75d5bac8d77ec8962f145fd6a7e68f"} Oct 30 21:17:40.680087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:40.680028 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-fpjcj" podStartSLOduration=1.9257104520000001 podStartE2EDuration="4.680012066s" totalImagesPullingTime="2.754301614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:40.678145141 +0000 UTC m=+374.660845560" watchObservedRunningTime="2026-10-30 21:17:40.680012066 +0000 UTC m=+374.662712484" Oct 30 21:17:43.321987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.321952 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vj998"] Oct 30 21:17:43.327131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.327109 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:43.382454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.382427 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hr28r\"" Oct 30 21:17:43.383705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.383682 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mws6s\" (UniqueName: \"kubernetes.io/projected/bc5307ad-9cdc-4c05-90e0-e7c43c1a628e-kube-api-access-mws6s\") pod \"authorino-operator-7b7bb4bc77-vj998\" (UID: \"bc5307ad-9cdc-4c05-90e0-e7c43c1a628e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:43.431379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.431352 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:17:43.431379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.431357 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:17:43.434792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.434768 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vj998"] Oct 30 21:17:43.484399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.484373 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws6s\" (UniqueName: \"kubernetes.io/projected/bc5307ad-9cdc-4c05-90e0-e7c43c1a628e-kube-api-access-mws6s\") pod \"authorino-operator-7b7bb4bc77-vj998\" (UID: \"bc5307ad-9cdc-4c05-90e0-e7c43c1a628e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:43.541946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.541911 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mws6s\" (UniqueName: \"kubernetes.io/projected/bc5307ad-9cdc-4c05-90e0-e7c43c1a628e-kube-api-access-mws6s\") pod \"authorino-operator-7b7bb4bc77-vj998\" (UID: \"bc5307ad-9cdc-4c05-90e0-e7c43c1a628e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:43.640305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:43.640220 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:44.017601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:44.017575 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vj998"] Oct 30 21:17:44.019261 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:44.019233 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5307ad_9cdc_4c05_90e0_e7c43c1a628e.slice/crio-dc8c3cdbf457863a0b6724f2af175a5990323726fb46e19643c362e5b41a93f8 WatchSource:0}: Error finding container dc8c3cdbf457863a0b6724f2af175a5990323726fb46e19643c362e5b41a93f8: Status 404 returned error can't find the container with id dc8c3cdbf457863a0b6724f2af175a5990323726fb46e19643c362e5b41a93f8 Oct 30 21:17:44.482929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:44.482879 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" event={"ID":"bc5307ad-9cdc-4c05-90e0-e7c43c1a628e","Type":"ContainerStarted","Data":"dc8c3cdbf457863a0b6724f2af175a5990323726fb46e19643c362e5b41a93f8"} Oct 30 21:17:47.108096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.108060 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm"] Oct 30 21:17:47.119685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.119657 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:47.171506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.171466 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm"] Oct 30 21:17:47.218020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.217979 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kdn79\"" Oct 30 21:17:47.219386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.219359 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wjq\" (UniqueName: \"kubernetes.io/projected/1bef682f-cb5d-48c3-a615-7fb2418d1ca8-kube-api-access-h8wjq\") pod \"limitador-operator-controller-manager-5778bb4449-xq4qm\" (UID: \"1bef682f-cb5d-48c3-a615-7fb2418d1ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:47.319948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.319889 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wjq\" (UniqueName: \"kubernetes.io/projected/1bef682f-cb5d-48c3-a615-7fb2418d1ca8-kube-api-access-h8wjq\") pod \"limitador-operator-controller-manager-5778bb4449-xq4qm\" (UID: \"1bef682f-cb5d-48c3-a615-7fb2418d1ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:47.377156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.377090 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wjq\" (UniqueName: \"kubernetes.io/projected/1bef682f-cb5d-48c3-a615-7fb2418d1ca8-kube-api-access-h8wjq\") pod \"limitador-operator-controller-manager-5778bb4449-xq4qm\" (UID: \"1bef682f-cb5d-48c3-a615-7fb2418d1ca8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:47.436173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.436137 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:47.815878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:47.815852 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm"] Oct 30 21:17:47.818100 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:47.818072 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef682f_cb5d_48c3_a615_7fb2418d1ca8.slice/crio-eba157ef8f0615198640c6f374160ff211ceb8ec329ca5b5cb6a9d716930c22e WatchSource:0}: Error finding container eba157ef8f0615198640c6f374160ff211ceb8ec329ca5b5cb6a9d716930c22e: Status 404 returned error can't find the container with id eba157ef8f0615198640c6f374160ff211ceb8ec329ca5b5cb6a9d716930c22e Oct 30 21:17:48.453712 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:48.453682 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" Oct 30 21:17:48.505546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:48.505508 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" event={"ID":"bc5307ad-9cdc-4c05-90e0-e7c43c1a628e","Type":"ContainerStarted","Data":"0b5afd868544ae3a89c2f23c90853319791fb8498141dbd334ca02e6fabb2c68"} Oct 30 21:17:48.505730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:48.505623 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:17:48.506824 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:48.506804 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" event={"ID":"1bef682f-cb5d-48c3-a615-7fb2418d1ca8","Type":"ContainerStarted","Data":"eba157ef8f0615198640c6f374160ff211ceb8ec329ca5b5cb6a9d716930c22e"} Oct 30 21:17:48.854447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:48.854337 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" podStartSLOduration=2.320092612 podStartE2EDuration="5.854307652s" totalImagesPullingTime="3.53421504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:48.85315848 +0000 UTC m=+382.835858900" watchObservedRunningTime="2026-10-30 21:17:48.854307652 +0000 UTC m=+382.837008070" Oct 30 21:17:51.529163 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:51.529117 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" event={"ID":"1bef682f-cb5d-48c3-a615-7fb2418d1ca8","Type":"ContainerStarted","Data":"721bccaeac61d5775a062046c3234b4aabdfdf4ece8eff7a020daa47ab7dcd30"} Oct 30 21:17:51.529664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:51.529248 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:17:51.647604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:51.647546 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" podStartSLOduration=2.001443903 podStartE2EDuration="4.647530569s" totalImagesPullingTime="2.646086666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:51.645158495 +0000 UTC m=+385.627858915" watchObservedRunningTime="2026-10-30 21:17:51.647530569 +0000 UTC m=+385.630230988" Oct 30 21:17:53.947308 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:53.947265 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:17:53.950980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:53.950958 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:53.997064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:53.997039 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:17:54.043878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.043843 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bkx8t\"" Oct 30 21:17:54.077239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.077204 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhs9w\" (UniqueName: \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.077394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.077355 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.178521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.178486 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhs9w\" (UniqueName: \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.178688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.178548 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.178927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.178909 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.236383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.236347 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhs9w\" (UniqueName: \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m94zs\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.263192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.263158 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:17:54.692169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:54.692145 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:17:54.694451 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:54.694420 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8955f17e_0b7d_4101_9f7b_847b0b1eef56.slice/crio-3dadf91bd39be21c1bb47964ec7b03c1fd60403de51cb9bc23f3469e5bc5a86b WatchSource:0}: Error finding container 3dadf91bd39be21c1bb47964ec7b03c1fd60403de51cb9bc23f3469e5bc5a86b: Status 404 returned error can't find the container with id 3dadf91bd39be21c1bb47964ec7b03c1fd60403de51cb9bc23f3469e5bc5a86b Oct 30 21:17:55.550280 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:55.550246 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" event={"ID":"8955f17e-0b7d-4101-9f7b-847b0b1eef56","Type":"ContainerStarted","Data":"3dadf91bd39be21c1bb47964ec7b03c1fd60403de51cb9bc23f3469e5bc5a86b"} Oct 30 21:17:57.728792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.728754 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8"] Oct 30 21:17:57.732178 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.732160 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:17:57.783710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.783678 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:17:57.814132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.814099 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjff\" (UniqueName: \"kubernetes.io/projected/7df10ec4-8bfc-44b6-98b1-5109f7380c48-kube-api-access-rbjff\") pod \"dns-operator-controller-manager-8557cbdb84-shhw8\" (UID: \"7df10ec4-8bfc-44b6-98b1-5109f7380c48\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:17:57.833048 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.833020 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-r8xhd\"" Oct 30 21:17:57.835428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.835402 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8"] Oct 30 21:17:57.915527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.915494 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjff\" (UniqueName: \"kubernetes.io/projected/7df10ec4-8bfc-44b6-98b1-5109f7380c48-kube-api-access-rbjff\") pod \"dns-operator-controller-manager-8557cbdb84-shhw8\" (UID: \"7df10ec4-8bfc-44b6-98b1-5109f7380c48\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:17:57.976610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:57.976576 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjff\" (UniqueName: \"kubernetes.io/projected/7df10ec4-8bfc-44b6-98b1-5109f7380c48-kube-api-access-rbjff\") pod \"dns-operator-controller-manager-8557cbdb84-shhw8\" (UID: \"7df10ec4-8bfc-44b6-98b1-5109f7380c48\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:17:58.045685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:58.045597 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:17:58.430209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:58.430184 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8"] Oct 30 21:17:58.433183 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:17:58.433141 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df10ec4_8bfc_44b6_98b1_5109f7380c48.slice/crio-215862129f91533c1988887339e051a8b2b7268a3028bdaba9c1954e81d93094 WatchSource:0}: Error finding container 215862129f91533c1988887339e051a8b2b7268a3028bdaba9c1954e81d93094: Status 404 returned error can't find the container with id 215862129f91533c1988887339e051a8b2b7268a3028bdaba9c1954e81d93094 Oct 30 21:17:58.569597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:58.569556 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" event={"ID":"7df10ec4-8bfc-44b6-98b1-5109f7380c48","Type":"ContainerStarted","Data":"215862129f91533c1988887339e051a8b2b7268a3028bdaba9c1954e81d93094"} Oct 30 21:17:59.515034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:17:59.515000 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vj998" Oct 30 21:18:00.584389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:00.584351 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" event={"ID":"8955f17e-0b7d-4101-9f7b-847b0b1eef56","Type":"ContainerStarted","Data":"9dbff9fd893b76d522bffeb897f638bb6098358d00628792954f81f4e6d7f3b7"} Oct 30 21:18:00.584857 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:00.584446 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:18:00.803904 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:00.803842 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" podStartSLOduration=2.839717544 podStartE2EDuration="7.803822176s" totalImagesPullingTime="4.964104632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:00.80336013 +0000 UTC m=+394.786060553" watchObservedRunningTime="2026-10-30 21:18:00.803822176 +0000 UTC m=+394.786522597" Oct 30 21:18:01.594096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:01.594006 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" event={"ID":"7df10ec4-8bfc-44b6-98b1-5109f7380c48","Type":"ContainerStarted","Data":"9c801f4ca7500cbf73ff6c5cf28e391da45a5133b821b28391dacf5abf0520fa"} Oct 30 21:18:01.594531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:01.594304 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:18:01.710847 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:01.710792 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" podStartSLOduration=1.881510686 podStartE2EDuration="4.7107779s" totalImagesPullingTime="2.829267214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:01.709058192 +0000 UTC m=+395.691758610" watchObservedRunningTime="2026-10-30 21:18:01.7107779 +0000 UTC m=+395.693478318" Oct 30 21:18:02.536621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:02.536586 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xq4qm" Oct 30 21:18:07.540091 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:18:07.540003 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:18:11.596653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:11.596617 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:18:11.932428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:11.932350 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 21:18:11.934740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:11.934724 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.000027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:11.999980 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 21:18:12.039430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.039399 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnln8\" (UniqueName: \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.039596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.039447 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.140487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.140454 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnln8\" (UniqueName: \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.140702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.140508 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.140902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.140885 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.197132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.197054 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnln8\" (UniqueName: \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\") pod \"kuadrant-operator-controller-manager-678d87766f-wvcpq\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.246626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.246594 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:12.602400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.602371 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-shhw8" Oct 30 21:18:12.715808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:12.715782 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 21:18:12.718286 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:18:12.718259 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb6e432_ebec_4cfa_bb6b_39badd83bb4e.slice/crio-d1b6ca54cb8092b6bcab8c2750348f88c37ab9283b6879b16fa907ba97b3cd20 WatchSource:0}: Error finding container d1b6ca54cb8092b6bcab8c2750348f88c37ab9283b6879b16fa907ba97b3cd20: Status 404 returned error can't find the container with id d1b6ca54cb8092b6bcab8c2750348f88c37ab9283b6879b16fa907ba97b3cd20 Oct 30 21:18:13.656599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:13.656557 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" event={"ID":"abb6e432-ebec-4cfa-bb6b-39badd83bb4e","Type":"ContainerStarted","Data":"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf"} Oct 30 21:18:13.657027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:13.656621 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" event={"ID":"abb6e432-ebec-4cfa-bb6b-39badd83bb4e","Type":"ContainerStarted","Data":"d1b6ca54cb8092b6bcab8c2750348f88c37ab9283b6879b16fa907ba97b3cd20"} Oct 30 21:18:13.657027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:13.656654 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:13.872231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:13.872165 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" podStartSLOduration=2.8721213309999998 podStartE2EDuration="2.872121331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:13.821386856 +0000 UTC m=+407.804087274" watchObservedRunningTime="2026-10-30 21:18:13.872121331 +0000 UTC m=+407.854821752" Oct 30 21:18:21.565440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.565404 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr"] Oct 30 21:18:21.568013 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.567997 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612291 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e7e234-f650-479c-83b9-26e9e9195867-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612382 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612412 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9x4b\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-kube-api-access-v9x4b\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612496 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612519 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612550 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612637 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e7e234-f650-479c-83b9-26e9e9195867-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612659 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.612692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.612678 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.619383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.619357 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jpjck\"" Oct 30 21:18:21.671172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.671128 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr"] Oct 30 21:18:21.713102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713068 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713107 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713143 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e7e234-f650-479c-83b9-26e9e9195867-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713172 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713206 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x4b\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-kube-api-access-v9x4b\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713245 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713268 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713301 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713388 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e7e234-f650-479c-83b9-26e9e9195867-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713561 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713617 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713669 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-data\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.713879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.713851 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.714103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.714086 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35e7e234-f650-479c-83b9-26e9e9195867-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.715780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.715756 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35e7e234-f650-479c-83b9-26e9e9195867-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.715963 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.715942 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35e7e234-f650-479c-83b9-26e9e9195867-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.769890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.769850 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9x4b\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-kube-api-access-v9x4b\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.770209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.770186 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35e7e234-f650-479c-83b9-26e9e9195867-istio-token\") pod \"data-science-gateway-data-science-gateway-class-549f9d654fqblwr\" (UID: \"35e7e234-f650-479c-83b9-26e9e9195867\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:21.882200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:21.882101 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:22.343450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.343341 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr"] Oct 30 21:18:22.346096 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:18:22.346041 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e7e234_f650_479c_83b9_26e9e9195867.slice/crio-08340b0816d1a772ff10840a5ab8acaaac3dd185a1b3a994448a98d3b5e51601 WatchSource:0}: Error finding container 08340b0816d1a772ff10840a5ab8acaaac3dd185a1b3a994448a98d3b5e51601: Status 404 returned error can't find the container with id 08340b0816d1a772ff10840a5ab8acaaac3dd185a1b3a994448a98d3b5e51601 Oct 30 21:18:22.348498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.348463 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:22.348592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.348543 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:22.348592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.348578 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:18:22.710833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.710794 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" event={"ID":"35e7e234-f650-479c-83b9-26e9e9195867","Type":"ContainerStarted","Data":"87cbd915535b3245fe32173e76c6398452b441e0fa3ed18336c9d4caaca710c0"} Oct 30 21:18:22.711260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.710860 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" event={"ID":"35e7e234-f650-479c-83b9-26e9e9195867","Type":"ContainerStarted","Data":"08340b0816d1a772ff10840a5ab8acaaac3dd185a1b3a994448a98d3b5e51601"} Oct 30 21:18:22.830351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.830272 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" podStartSLOduration=1.830256525 podStartE2EDuration="1.830256525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:22.828292165 +0000 UTC m=+416.810992584" watchObservedRunningTime="2026-10-30 21:18:22.830256525 +0000 UTC m=+416.812956943" Oct 30 21:18:22.883149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:22.883113 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:23.888761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:23.888731 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:24.664922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:24.664891 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 21:18:24.721877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:24.721847 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:24.722825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:24.722806 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-549f9d654fqblwr" Oct 30 21:18:24.885056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:24.885033 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:18:24.885281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:24.885259 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" podUID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" containerName="manager" containerID="cri-o://9dbff9fd893b76d522bffeb897f638bb6098358d00628792954f81f4e6d7f3b7" gracePeriod=10 Oct 30 21:18:26.737511 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.737482 4921 generic.go:356] "Generic (PLEG): container finished" podID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" containerID="9dbff9fd893b76d522bffeb897f638bb6098358d00628792954f81f4e6d7f3b7" exitCode=0 Oct 30 21:18:26.737948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.737538 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" event={"ID":"8955f17e-0b7d-4101-9f7b-847b0b1eef56","Type":"ContainerDied","Data":"9dbff9fd893b76d522bffeb897f638bb6098358d00628792954f81f4e6d7f3b7"} Oct 30 21:18:26.823061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.823040 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:18:26.851505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.851481 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\") pod \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " Oct 30 21:18:26.851837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.851805 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume" pod "8955f17e-0b7d-4101-9f7b-847b0b1eef56" (UID: "8955f17e-0b7d-4101-9f7b-847b0b1eef56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:26.952434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.952414 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\" (UniqueName: \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\") pod \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\" (UID: \"8955f17e-0b7d-4101-9f7b-847b0b1eef56\") " Oct 30 21:18:26.952554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.952540 4921 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8955f17e-0b7d-4101-9f7b-847b0b1eef56-extensions-socket-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:26.954337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:26.954303 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w" pod "8955f17e-0b7d-4101-9f7b-847b0b1eef56" (UID: "8955f17e-0b7d-4101-9f7b-847b0b1eef56"). InnerVolumeSpecName "kube-api-access-qhs9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:27.053438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.053412 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhs9w\" (UniqueName: \"kubernetes.io/projected/8955f17e-0b7d-4101-9f7b-847b0b1eef56-kube-api-access-qhs9w\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:27.744169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.744138 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" Oct 30 21:18:27.744674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.744138 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs" event={"ID":"8955f17e-0b7d-4101-9f7b-847b0b1eef56","Type":"ContainerDied","Data":"3dadf91bd39be21c1bb47964ec7b03c1fd60403de51cb9bc23f3469e5bc5a86b"} Oct 30 21:18:27.744674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.744286 4921 scope.go:122] "RemoveContainer" containerID="9dbff9fd893b76d522bffeb897f638bb6098358d00628792954f81f4e6d7f3b7" Oct 30 21:18:27.905670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.905633 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:18:27.955439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:27.955410 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m94zs"] Oct 30 21:18:28.166504 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:28.166434 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" path="/var/lib/kubelet/pods/8955f17e-0b7d-4101-9f7b-847b0b1eef56/volumes" Oct 30 21:18:33.371429 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.371399 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9"] Oct 30 21:18:33.371792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.371727 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" containerName="manager" Oct 30 21:18:33.371792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.371737 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" containerName="manager" Oct 30 21:18:33.371860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.371825 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8955f17e-0b7d-4101-9f7b-847b0b1eef56" containerName="manager" Oct 30 21:18:33.403004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.402978 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.434805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.434770 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9"] Oct 30 21:18:33.480844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.480821 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-hsf7l\"" Oct 30 21:18:33.481615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.481598 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:18:33.481615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.481610 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:18:33.497526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.497507 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.497612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.497540 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.497612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.497574 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.497708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.497661 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjq9z\" (UniqueName: \"kubernetes.io/projected/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-kube-api-access-jjq9z\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.580686 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.580667 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:18:33.598186 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598168 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598203 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598231 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598272 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjq9z\" (UniqueName: \"kubernetes.io/projected/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-kube-api-access-jjq9z\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598518 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598582 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598567 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.598989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.598973 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.654733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.654688 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjq9z\" (UniqueName: \"kubernetes.io/projected/4ba4bda3-21eb-4b27-9c77-2cc7c05707c9-kube-api-access-jjq9z\") pod \"kserve-module-controller-manager-7566cf5fc8-fw5z9\" (UID: \"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:33.729280 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:33.729252 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:34.174382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:34.174352 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9"] Oct 30 21:18:34.214023 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:18:34.213997 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4bda3_21eb_4b27_9c77_2cc7c05707c9.slice/crio-78e1a67e91d084cd6cef894163abf2e516506a1fd1e62183c64e650b137af484 WatchSource:0}: Error finding container 78e1a67e91d084cd6cef894163abf2e516506a1fd1e62183c64e650b137af484: Status 404 returned error can't find the container with id 78e1a67e91d084cd6cef894163abf2e516506a1fd1e62183c64e650b137af484 Oct 30 21:18:34.782013 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:34.781983 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" event={"ID":"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9","Type":"ContainerStarted","Data":"78e1a67e91d084cd6cef894163abf2e516506a1fd1e62183c64e650b137af484"} Oct 30 21:18:37.538356 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:18:37.538306 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:18:37.799002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:37.798926 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" event={"ID":"4ba4bda3-21eb-4b27-9c77-2cc7c05707c9","Type":"ContainerStarted","Data":"0678852bb24e73e9580cca9eac0ecdba6efe07f93d927d24e045f9d0989d7ac5"} Oct 30 21:18:37.799149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:37.799075 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:18:38.016015 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:38.015967 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" podStartSLOduration=2.473985972 podStartE2EDuration="5.015954993s" totalImagesPullingTime="2.541969021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:37.965244058 +0000 UTC m=+431.947944476" watchObservedRunningTime="2026-10-30 21:18:38.015954993 +0000 UTC m=+431.998655411" Oct 30 21:18:46.505710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.505678 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:18:46.507959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.507942 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:18:46.564918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.564893 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:18:46.605470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.605445 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qps\" (UniqueName: \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\") pod \"authorino-56c6c9ddd5-qvwhq\" (UID: \"c4e7e440-71b9-42f0-84ee-6027442e75a2\") " pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:18:46.611715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.611695 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvcbr\"" Oct 30 21:18:46.706697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.706672 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qps\" (UniqueName: \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\") pod \"authorino-56c6c9ddd5-qvwhq\" (UID: \"c4e7e440-71b9-42f0-84ee-6027442e75a2\") " pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:18:46.762041 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.761986 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qps\" (UniqueName: \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\") pod \"authorino-56c6c9ddd5-qvwhq\" (UID: \"c4e7e440-71b9-42f0-84ee-6027442e75a2\") " pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:18:46.826487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:46.826466 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:18:47.198342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:47.198300 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:18:47.200246 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:18:47.200220 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4e7e440_71b9_42f0_84ee_6027442e75a2.slice/crio-a2416ce5bc3d042500bf0267fa5d0211a2a64075a882fbfebec8844f6a6c12b4 WatchSource:0}: Error finding container a2416ce5bc3d042500bf0267fa5d0211a2a64075a882fbfebec8844f6a6c12b4: Status 404 returned error can't find the container with id a2416ce5bc3d042500bf0267fa5d0211a2a64075a882fbfebec8844f6a6c12b4 Oct 30 21:18:47.852056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:47.852016 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" event={"ID":"c4e7e440-71b9-42f0-84ee-6027442e75a2","Type":"ContainerStarted","Data":"a2416ce5bc3d042500bf0267fa5d0211a2a64075a882fbfebec8844f6a6c12b4"} Oct 30 21:18:49.865548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:49.865511 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" event={"ID":"c4e7e440-71b9-42f0-84ee-6027442e75a2","Type":"ContainerStarted","Data":"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde"} Oct 30 21:18:49.986827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:49.986776 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" podStartSLOduration=1.835803477 podStartE2EDuration="3.986762573s" totalImagesPullingTime="2.150959096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:49.985612977 +0000 UTC m=+443.968313396" watchObservedRunningTime="2026-10-30 21:18:49.986762573 +0000 UTC m=+443.969462996" Oct 30 21:18:58.805625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:18:58.805596 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fw5z9" Oct 30 21:19:02.672116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.672084 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw"] Oct 30 21:19:02.674859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.674841 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.726477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.726452 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:19:02.746733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.746707 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvml6\" (UniqueName: \"kubernetes.io/projected/cef2e5c6-83a4-4880-b42d-90630d6bd804-kube-api-access-bvml6\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.746853 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.746818 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cef2e5c6-83a4-4880-b42d-90630d6bd804-data\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.765555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.765534 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:19:02.765555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.765538 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:19:02.775886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.775868 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wfkcx\"" Oct 30 21:19:02.778032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.778003 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw"] Oct 30 21:19:02.847887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.847859 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvml6\" (UniqueName: \"kubernetes.io/projected/cef2e5c6-83a4-4880-b42d-90630d6bd804-kube-api-access-bvml6\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.848025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.848005 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cef2e5c6-83a4-4880-b42d-90630d6bd804-data\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.848390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.848373 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cef2e5c6-83a4-4880-b42d-90630d6bd804-data\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.907196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.907166 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvml6\" (UniqueName: \"kubernetes.io/projected/cef2e5c6-83a4-4880-b42d-90630d6bd804-kube-api-access-bvml6\") pod \"postgres-7cdbf6dc44-zbgnw\" (UID: \"cef2e5c6-83a4-4880-b42d-90630d6bd804\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:02.985403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:02.985364 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:03.354605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:03.354577 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw"] Oct 30 21:19:03.356245 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:19:03.356219 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef2e5c6_83a4_4880_b42d_90630d6bd804.slice/crio-7e2785234bf6301d29bdc64fdf9bed9f848959ac87d0984d3aab0b6be680f702 WatchSource:0}: Error finding container 7e2785234bf6301d29bdc64fdf9bed9f848959ac87d0984d3aab0b6be680f702: Status 404 returned error can't find the container with id 7e2785234bf6301d29bdc64fdf9bed9f848959ac87d0984d3aab0b6be680f702 Oct 30 21:19:03.936803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:03.936773 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" event={"ID":"cef2e5c6-83a4-4880-b42d-90630d6bd804","Type":"ContainerStarted","Data":"7e2785234bf6301d29bdc64fdf9bed9f848959ac87d0984d3aab0b6be680f702"} Oct 30 21:19:07.543877 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:19:07.543839 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:19:10.981569 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:10.981538 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" event={"ID":"cef2e5c6-83a4-4880-b42d-90630d6bd804","Type":"ContainerStarted","Data":"eafe2f835ae02b1501b4d95bbd78005523bd6a33d9fa4a16cb410595984e4d28"} Oct 30 21:19:10.981929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:10.981638 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:11.096010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:11.095964 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" podStartSLOduration=2.123141217 podStartE2EDuration="9.095951291s" totalImagesPullingTime="6.972810074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:11.094298689 +0000 UTC m=+465.076999108" watchObservedRunningTime="2026-10-30 21:19:11.095951291 +0000 UTC m=+465.078651709" Oct 30 21:19:17.015517 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:17.015489 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zbgnw" Oct 30 21:19:37.538801 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:19:37.538723 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:19:38.800619 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.800589 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-nl2n2"] Oct 30 21:19:38.804203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.804180 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.835719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.835692 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4g94\" (UniqueName: \"kubernetes.io/projected/dcf552bb-d943-4039-9f25-fd35df68f826-kube-api-access-l4g94\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.835719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.835723 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf552bb-d943-4039-9f25-fd35df68f826-cert\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.861713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.861688 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-nl2n2"] Oct 30 21:19:38.899562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.899534 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:19:38.899562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.899534 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-59ddk\"" Oct 30 21:19:38.936266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.936235 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4g94\" (UniqueName: \"kubernetes.io/projected/dcf552bb-d943-4039-9f25-fd35df68f826-kube-api-access-l4g94\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.936430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.936272 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf552bb-d943-4039-9f25-fd35df68f826-cert\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.938855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.938835 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf552bb-d943-4039-9f25-fd35df68f826-cert\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:38.992309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:38.992285 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4g94\" (UniqueName: \"kubernetes.io/projected/dcf552bb-d943-4039-9f25-fd35df68f826-kube-api-access-l4g94\") pod \"kserve-controller-manager-776cc66d47-nl2n2\" (UID: \"dcf552bb-d943-4039-9f25-fd35df68f826\") " pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:39.028090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.028065 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vlp2m"] Oct 30 21:19:39.033730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.033703 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.086534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.086411 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hk64h\"" Oct 30 21:19:39.086534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.086437 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:19:39.089104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.089082 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vlp2m"] Oct 30 21:19:39.118824 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.118805 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:39.137779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.137760 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwzc\" (UniqueName: \"kubernetes.io/projected/74f023e8-abb9-482b-9597-f4e9e60831d8-kube-api-access-dqwzc\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.137876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.137804 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f023e8-abb9-482b-9597-f4e9e60831d8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.238790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.238760 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwzc\" (UniqueName: \"kubernetes.io/projected/74f023e8-abb9-482b-9597-f4e9e60831d8-kube-api-access-dqwzc\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.238959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.238939 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f023e8-abb9-482b-9597-f4e9e60831d8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.241092 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.241073 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f023e8-abb9-482b-9597-f4e9e60831d8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.295238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.295216 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwzc\" (UniqueName: \"kubernetes.io/projected/74f023e8-abb9-482b-9597-f4e9e60831d8-kube-api-access-dqwzc\") pod \"model-serving-api-75bd89fdfc-vlp2m\" (UID: \"74f023e8-abb9-482b-9597-f4e9e60831d8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.346624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.346579 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:39.490794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.490771 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-nl2n2"] Oct 30 21:19:39.492155 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:19:39.492127 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf552bb_d943_4039_9f25_fd35df68f826.slice/crio-e8d044b7f0e880eb5e6d098db84caa76fcac8339dccc407bdfb83914d4a7c2eb WatchSource:0}: Error finding container e8d044b7f0e880eb5e6d098db84caa76fcac8339dccc407bdfb83914d4a7c2eb: Status 404 returned error can't find the container with id e8d044b7f0e880eb5e6d098db84caa76fcac8339dccc407bdfb83914d4a7c2eb Oct 30 21:19:39.715988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:39.715966 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vlp2m"] Oct 30 21:19:39.717820 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:19:39.717795 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f023e8_abb9_482b_9597_f4e9e60831d8.slice/crio-a0f3a65de04a1f24623d7e1e94e2aebb40f424bfe95ecdb47b957b7c84df38c7 WatchSource:0}: Error finding container a0f3a65de04a1f24623d7e1e94e2aebb40f424bfe95ecdb47b957b7c84df38c7: Status 404 returned error can't find the container with id a0f3a65de04a1f24623d7e1e94e2aebb40f424bfe95ecdb47b957b7c84df38c7 Oct 30 21:19:40.134560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:40.134493 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" event={"ID":"dcf552bb-d943-4039-9f25-fd35df68f826","Type":"ContainerStarted","Data":"e8d044b7f0e880eb5e6d098db84caa76fcac8339dccc407bdfb83914d4a7c2eb"} Oct 30 21:19:40.135657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:40.135632 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" event={"ID":"74f023e8-abb9-482b-9597-f4e9e60831d8","Type":"ContainerStarted","Data":"a0f3a65de04a1f24623d7e1e94e2aebb40f424bfe95ecdb47b957b7c84df38c7"} Oct 30 21:19:43.159004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.158964 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" event={"ID":"dcf552bb-d943-4039-9f25-fd35df68f826","Type":"ContainerStarted","Data":"84830270026b2106fd8e69f66af091c251da26e053db5a3997f9684bafaba033"} Oct 30 21:19:43.159578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.159031 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:19:43.160646 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.160621 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" event={"ID":"74f023e8-abb9-482b-9597-f4e9e60831d8","Type":"ContainerStarted","Data":"f4aff1a9d97ab0e7a071d86565d4f4128ce99da7ea483381c590ceda41785875"} Oct 30 21:19:43.160752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.160672 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:19:43.273302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.273258 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" podStartSLOduration=2.628583368 podStartE2EDuration="5.273246134s" totalImagesPullingTime="2.644662766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:43.272114966 +0000 UTC m=+497.254815383" watchObservedRunningTime="2026-10-30 21:19:43.273246134 +0000 UTC m=+497.255946551" Oct 30 21:19:43.387821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:43.387774 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" podStartSLOduration=2.924754712 podStartE2EDuration="5.387759897s" totalImagesPullingTime="2.463005185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:43.385518251 +0000 UTC m=+497.368218651" watchObservedRunningTime="2026-10-30 21:19:43.387759897 +0000 UTC m=+497.370460313" Oct 30 21:19:54.168756 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:19:54.168730 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vlp2m" Oct 30 21:20:07.538859 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:20:07.538824 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:20:14.170780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:14.170753 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-nl2n2" Oct 30 21:20:18.683501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:18.683457 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:20:18.683874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:18.683526 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:20:19.360566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.360506 4921 generic.go:356] "Generic (PLEG): container finished" podID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerID="4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005" exitCode=0 Oct 30 21:20:19.360712 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.360575 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerDied","Data":"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005"} Oct 30 21:20:19.360892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.360878 4921 scope.go:122] "RemoveContainer" containerID="4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005" Oct 30 21:20:19.361884 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.361858 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:19.361934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.361924 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:19.362003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:19.361990 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:20:20.367653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:20.367617 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerStarted","Data":"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb"} Oct 30 21:20:21.159839 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:21.159806 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:20:21.160020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:21.160001 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:20:21.160067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:21.160052 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 21:20:22.159813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:22.159782 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:20:22.160181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:22.159829 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 21:20:23.159464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:23.159430 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:20:23.159632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:23.159482 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 21:20:24.160055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:24.160023 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Oct 30 21:20:24.160433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:24.160075 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Oct 30 21:20:25.164116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:25.164087 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:20:25.164684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:25.164537 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:20:25.165333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:25.165285 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:20:35.854902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:35.854869 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:35.858011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:35.857996 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:35.915620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:35.915595 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:35.919231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:35.919210 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwr5m\" (UniqueName: \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\") pod \"authorino-6f6944bcdd-wpwqq\" (UID: \"52b6ee86-6fcd-414f-a1d5-ef0b2848b534\") " pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:36.019688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:36.019666 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwr5m\" (UniqueName: \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\") pod \"authorino-6f6944bcdd-wpwqq\" (UID: \"52b6ee86-6fcd-414f-a1d5-ef0b2848b534\") " pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:36.076275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:36.076247 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwr5m\" (UniqueName: \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\") pod \"authorino-6f6944bcdd-wpwqq\" (UID: \"52b6ee86-6fcd-414f-a1d5-ef0b2848b534\") " pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:36.168579 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:36.168522 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:36.217781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:36.217753 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:36.534798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:36.534777 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:36.536650 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:20:36.536624 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52b6ee86_6fcd_414f_a1d5_ef0b2848b534.slice/crio-0e71462c940223d0d12eed5e6fa2d46ffa8c9c85003719c569568eeef099831c WatchSource:0}: Error finding container 0e71462c940223d0d12eed5e6fa2d46ffa8c9c85003719c569568eeef099831c: Status 404 returned error can't find the container with id 0e71462c940223d0d12eed5e6fa2d46ffa8c9c85003719c569568eeef099831c Oct 30 21:20:37.456302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:37.456267 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" event={"ID":"52b6ee86-6fcd-414f-a1d5-ef0b2848b534","Type":"ContainerStarted","Data":"0e71462c940223d0d12eed5e6fa2d46ffa8c9c85003719c569568eeef099831c"} Oct 30 21:20:37.538094 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:20:37.538065 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:20:38.463709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.463676 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" event={"ID":"52b6ee86-6fcd-414f-a1d5-ef0b2848b534","Type":"ContainerStarted","Data":"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c"} Oct 30 21:20:38.464171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.463761 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" podUID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" containerName="authorino" containerID="cri-o://714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c" gracePeriod=30 Oct 30 21:20:38.576539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.576490 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" podStartSLOduration=2.105412551 podStartE2EDuration="3.57647739s" totalImagesPullingTime="1.471064839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:38.574595667 +0000 UTC m=+552.557296078" watchObservedRunningTime="2026-10-30 21:20:38.57647739 +0000 UTC m=+552.559177808" Oct 30 21:20:38.704545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.704514 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:38.739926 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.739875 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\" (UniqueName: \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\") pod \"52b6ee86-6fcd-414f-a1d5-ef0b2848b534\" (UID: \"52b6ee86-6fcd-414f-a1d5-ef0b2848b534\") " Oct 30 21:20:38.741748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.741725 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m" pod "52b6ee86-6fcd-414f-a1d5-ef0b2848b534" (UID: "52b6ee86-6fcd-414f-a1d5-ef0b2848b534"). InnerVolumeSpecName "kube-api-access-nwr5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:38.841113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:38.841078 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwr5m\" (UniqueName: \"kubernetes.io/projected/52b6ee86-6fcd-414f-a1d5-ef0b2848b534-kube-api-access-nwr5m\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:39.478242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.478206 4921 generic.go:356] "Generic (PLEG): container finished" podID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" containerID="714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c" exitCode=0 Oct 30 21:20:39.478695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.478246 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" Oct 30 21:20:39.478695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.478306 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" event={"ID":"52b6ee86-6fcd-414f-a1d5-ef0b2848b534","Type":"ContainerDied","Data":"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c"} Oct 30 21:20:39.478695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.478357 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wpwqq" event={"ID":"52b6ee86-6fcd-414f-a1d5-ef0b2848b534","Type":"ContainerDied","Data":"0e71462c940223d0d12eed5e6fa2d46ffa8c9c85003719c569568eeef099831c"} Oct 30 21:20:39.478695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.478378 4921 scope.go:122] "RemoveContainer" containerID="714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c" Oct 30 21:20:39.490024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.490007 4921 scope.go:122] "RemoveContainer" containerID="714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c" Oct 30 21:20:39.490284 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:20:39.490266 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c\": container with ID starting with 714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c not found: ID does not exist" containerID="714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c" Oct 30 21:20:39.490339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.490292 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c"} err="failed to get container status \"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c\": rpc error: code = NotFound desc = could not find container \"714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c\": container with ID starting with 714cde347b32e83ebad8fd8f240a7a2202c96701fbca6d644633adf43a95d01c not found: ID does not exist" Oct 30 21:20:39.641945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.641910 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:39.645083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.645058 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wpwqq"] Oct 30 21:20:39.858981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.858907 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:20:39.859170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:39.859092 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" podUID="c4e7e440-71b9-42f0-84ee-6027442e75a2" containerName="authorino" containerID="cri-o://b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde" gracePeriod=30 Oct 30 21:20:40.096931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.096909 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:20:40.151722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.151674 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\" (UniqueName: \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\") pod \"c4e7e440-71b9-42f0-84ee-6027442e75a2\" (UID: \"c4e7e440-71b9-42f0-84ee-6027442e75a2\") " Oct 30 21:20:40.153540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.153512 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps" pod "c4e7e440-71b9-42f0-84ee-6027442e75a2" (UID: "c4e7e440-71b9-42f0-84ee-6027442e75a2"). InnerVolumeSpecName "kube-api-access-k6qps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:40.166406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.166384 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" path="/var/lib/kubelet/pods/52b6ee86-6fcd-414f-a1d5-ef0b2848b534/volumes" Oct 30 21:20:40.252531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.252508 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qps\" (UniqueName: \"kubernetes.io/projected/c4e7e440-71b9-42f0-84ee-6027442e75a2-kube-api-access-k6qps\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:20:40.485574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.485547 4921 generic.go:356] "Generic (PLEG): container finished" podID="c4e7e440-71b9-42f0-84ee-6027442e75a2" containerID="b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde" exitCode=0 Oct 30 21:20:40.486027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.485584 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" Oct 30 21:20:40.486027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.485627 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" event={"ID":"c4e7e440-71b9-42f0-84ee-6027442e75a2","Type":"ContainerDied","Data":"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde"} Oct 30 21:20:40.486027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.485662 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qvwhq" event={"ID":"c4e7e440-71b9-42f0-84ee-6027442e75a2","Type":"ContainerDied","Data":"a2416ce5bc3d042500bf0267fa5d0211a2a64075a882fbfebec8844f6a6c12b4"} Oct 30 21:20:40.486027 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.485678 4921 scope.go:122] "RemoveContainer" containerID="b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde" Oct 30 21:20:40.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.493828 4921 scope.go:122] "RemoveContainer" containerID="b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde" Oct 30 21:20:40.494075 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:20:40.494057 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde\": container with ID starting with b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde not found: ID does not exist" containerID="b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde" Oct 30 21:20:40.494132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.494088 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde"} err="failed to get container status \"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde\": rpc error: code = NotFound desc = could not find container \"b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde\": container with ID starting with b37fc6f3459a7704b1bbc7426797a8a3ad0515321c4c5c3366ba23f518d1edde not found: ID does not exist" Oct 30 21:20:40.646902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.646876 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:20:40.650202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:40.650184 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qvwhq"] Oct 30 21:20:42.165921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:20:42.165894 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e7e440-71b9-42f0-84ee-6027442e75a2" path="/var/lib/kubelet/pods/c4e7e440-71b9-42f0-84ee-6027442e75a2/volumes" Oct 30 21:21:07.538543 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:21:07.538462 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:21:26.198111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:26.198081 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:21:26.198839 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:26.198818 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:21:26.208667 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:26.208647 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:21:26.209351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:26.209337 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:21:37.538310 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:21:37.538274 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:21:56.700017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.699979 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q"] Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700435 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c4e7e440-71b9-42f0-84ee-6027442e75a2" containerName="authorino" Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700448 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e7e440-71b9-42f0-84ee-6027442e75a2" containerName="authorino" Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700476 4921 cpu_manager.go:414] "containerMap: removing container" podUID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" containerName="authorino" Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700481 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" containerName="authorino" Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700543 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b6ee86-6fcd-414f-a1d5-ef0b2848b534" containerName="authorino" Oct 30 21:21:56.700574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.700555 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e7e440-71b9-42f0-84ee-6027442e75a2" containerName="authorino" Oct 30 21:21:56.703533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.703509 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725022 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.724987 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.725032 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.725191 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5mg\" (UniqueName: \"kubernetes.io/projected/659904f4-f858-4f39-8269-dd124fb116b7-kube-api-access-6x5mg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.725260 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.725291 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659904f4-f858-4f39-8269-dd124fb116b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.725416 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.725395 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.755699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.755670 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:21:56.804878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.804835 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:21:56.804878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.804856 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cpqws\"" Oct 30 21:21:56.805097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.804902 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:21:56.807176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.807154 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q"] Oct 30 21:21:56.825874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825840 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.825874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825875 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825897 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5mg\" (UniqueName: \"kubernetes.io/projected/659904f4-f858-4f39-8269-dd124fb116b7-kube-api-access-6x5mg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825931 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825945 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659904f4-f858-4f39-8269-dd124fb116b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.825967 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.826244 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.826307 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.826505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.826418 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.828269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.828249 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/659904f4-f858-4f39-8269-dd124fb116b7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.828451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.828436 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/659904f4-f858-4f39-8269-dd124fb116b7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:56.882394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:56.882358 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5mg\" (UniqueName: \"kubernetes.io/projected/659904f4-f858-4f39-8269-dd124fb116b7-kube-api-access-6x5mg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q\" (UID: \"659904f4-f858-4f39-8269-dd124fb116b7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:57.016542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:57.016459 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:21:57.389229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:57.389204 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q"] Oct 30 21:21:57.391116 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:21:57.391074 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659904f4_f858_4f39_8269_dd124fb116b7.slice/crio-a381567cc6fbfed773d93895fde037d551f0af284a899387935286401a456a55 WatchSource:0}: Error finding container a381567cc6fbfed773d93895fde037d551f0af284a899387935286401a456a55: Status 404 returned error can't find the container with id a381567cc6fbfed773d93895fde037d551f0af284a899387935286401a456a55 Oct 30 21:21:57.392438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:57.392420 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:21:57.869046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:57.869007 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" event={"ID":"659904f4-f858-4f39-8269-dd124fb116b7","Type":"ContainerStarted","Data":"a381567cc6fbfed773d93895fde037d551f0af284a899387935286401a456a55"} Oct 30 21:21:59.886195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:21:59.886164 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" event={"ID":"659904f4-f858-4f39-8269-dd124fb116b7","Type":"ContainerStarted","Data":"e818b8872ff50f8bcbb579e9704378f1151822110dac0d43897206a817b89eff"} Oct 30 21:22:00.002882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:00.002819 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" podStartSLOduration=2.110371242 podStartE2EDuration="4.002610117s" totalImagesPullingTime="1.892238875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:00.002148889 +0000 UTC m=+633.984849310" watchObservedRunningTime="2026-10-30 21:22:00.002610117 +0000 UTC m=+633.985310538" Oct 30 21:22:00.017237 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:00.017204 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:22:01.041838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.041797 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rv945"] Oct 30 21:22:01.045540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.045521 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059078 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-home\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059116 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059134 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059183 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e68b799-38c3-4d0d-837a-4587ac587861-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059296 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hfv\" (UniqueName: \"kubernetes.io/projected/4e68b799-38c3-4d0d-837a-4587ac587861-kube-api-access-x7hfv\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.059408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.059380 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.120328 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.120287 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:22:01.149228 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.149189 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rv945"] Oct 30 21:22:01.159816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.159788 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-home\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.159840 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.159913 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.159938 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e68b799-38c3-4d0d-837a-4587ac587861-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.160004 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hfv\" (UniqueName: \"kubernetes.io/projected/4e68b799-38c3-4d0d-837a-4587ac587861-kube-api-access-x7hfv\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.160073 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160627 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.160604 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.160659 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-home\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.160703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.160668 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.162545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.162528 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e68b799-38c3-4d0d-837a-4587ac587861-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.162724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.162709 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e68b799-38c3-4d0d-837a-4587ac587861-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.216680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.216647 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hfv\" (UniqueName: \"kubernetes.io/projected/4e68b799-38c3-4d0d-837a-4587ac587861-kube-api-access-x7hfv\") pod \"echo-simulator-kserve-64b6f8f96c-rv945\" (UID: \"4e68b799-38c3-4d0d-837a-4587ac587861\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.358125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.358033 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:01.734675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.734644 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rv945"] Oct 30 21:22:01.736350 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:01.736323 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e68b799_38c3_4d0d_837a_4587ac587861.slice/crio-ab15238dbc8a49c19de61675760e813e565b3ba8239661e46a645fb73597fba4 WatchSource:0}: Error finding container ab15238dbc8a49c19de61675760e813e565b3ba8239661e46a645fb73597fba4: Status 404 returned error can't find the container with id ab15238dbc8a49c19de61675760e813e565b3ba8239661e46a645fb73597fba4 Oct 30 21:22:01.897842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:01.897795 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" event={"ID":"4e68b799-38c3-4d0d-837a-4587ac587861","Type":"ContainerStarted","Data":"ab15238dbc8a49c19de61675760e813e565b3ba8239661e46a645fb73597fba4"} Oct 30 21:22:04.179822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.179786 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs"] Oct 30 21:22:04.184062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.184038 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.241525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.241486 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs"] Oct 30 21:22:04.286486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.286453 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:22:04.295069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295033 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.295261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295091 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.295261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295162 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.295261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295237 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.295438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295278 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc79g\" (UniqueName: \"kubernetes.io/projected/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-kube-api-access-zc79g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.295438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.295325 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.395977 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc79g\" (UniqueName: \"kubernetes.io/projected/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-kube-api-access-zc79g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396027 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396087 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396120 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396146 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396181 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396604 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396622 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.396750 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.396697 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.398928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.398895 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.399064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.399043 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.455106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.455076 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc79g\" (UniqueName: \"kubernetes.io/projected/2eb29d8b-7e33-4068-9fd5-8ceb7cde6606-kube-api-access-zc79g\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs\" (UID: \"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.500039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.499996 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:04.889095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.889065 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs"] Oct 30 21:22:04.890661 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:04.890634 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb29d8b_7e33_4068_9fd5_8ceb7cde6606.slice/crio-50422020d75b838df94f9cf52bf6c96f51d035dd73242ae05589b11fba79ea60 WatchSource:0}: Error finding container 50422020d75b838df94f9cf52bf6c96f51d035dd73242ae05589b11fba79ea60: Status 404 returned error can't find the container with id 50422020d75b838df94f9cf52bf6c96f51d035dd73242ae05589b11fba79ea60 Oct 30 21:22:04.924256 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.924218 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" event={"ID":"4e68b799-38c3-4d0d-837a-4587ac587861","Type":"ContainerStarted","Data":"025a16c95a54c39ed01b19478b5447c4b7fda66ddf9651a9ae8562e490916dcd"} Oct 30 21:22:04.926097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:04.926067 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" event={"ID":"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606","Type":"ContainerStarted","Data":"50422020d75b838df94f9cf52bf6c96f51d035dd73242ae05589b11fba79ea60"} Oct 30 21:22:05.039645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:05.039594 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" podStartSLOduration=1.406056509 podStartE2EDuration="4.03958024s" totalImagesPullingTime="2.633523731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:05.037630062 +0000 UTC m=+639.020330483" watchObservedRunningTime="2026-10-30 21:22:05.03958024 +0000 UTC m=+639.022280656" Oct 30 21:22:05.358636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:05.358588 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:05.933029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:05.932991 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" event={"ID":"2eb29d8b-7e33-4068-9fd5-8ceb7cde6606","Type":"ContainerStarted","Data":"ce350c302d84f771190ca26c8b55e83d0416717f6c64e402c983fc5e49427691"} Oct 30 21:22:06.048360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:06.048289 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" podStartSLOduration=1.40585701 podStartE2EDuration="2.048275586s" totalImagesPullingTime="642.418576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:06.046369172 +0000 UTC m=+640.029069588" watchObservedRunningTime="2026-10-30 21:22:06.048275586 +0000 UTC m=+640.030976004" Oct 30 21:22:06.500656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:06.500617 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:07.016622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:07.016590 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:22:07.029817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:07.029782 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:22:07.540823 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:22:07.540782 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:22:07.957300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:07.957268 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q" Oct 30 21:22:11.358871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:11.358833 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:11.371610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:11.371580 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:11.986111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:11.986067 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rv945" Oct 30 21:22:14.500606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:14.500568 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:14.513213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:14.513182 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:14.999015 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:14.998985 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs" Oct 30 21:22:17.774171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.774119 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj"] Oct 30 21:22:17.780585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.780554 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.882213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.882171 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:22:17.884799 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.884766 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj"] Oct 30 21:22:17.935415 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935380 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.935601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935445 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.935601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935497 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.935694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935603 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.935694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935619 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:17.935694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:17.935641 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sx5\" (UniqueName: \"kubernetes.io/projected/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-kube-api-access-s7sx5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036086 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036137 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036207 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036228 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036258 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sx5\" (UniqueName: \"kubernetes.io/projected/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-kube-api-access-s7sx5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036346 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036751 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036854 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036789 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.036990 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.036931 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.039054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.039026 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.039550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.039525 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.093365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.093310 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sx5\" (UniqueName: \"kubernetes.io/projected/1a3e99ee-1a34-4531-99e4-5a7a083bce3b-kube-api-access-s7sx5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-l7xtj\" (UID: \"1a3e99ee-1a34-4531-99e4-5a7a083bce3b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.099050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.099009 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:18.497665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:18.497636 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj"] Oct 30 21:22:18.499187 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:18.499158 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3e99ee_1a34_4531_99e4_5a7a083bce3b.slice/crio-15d236f9584e5c4ea80af2baf07cc872e716f93e11694e868bb9adaabfa6b647 WatchSource:0}: Error finding container 15d236f9584e5c4ea80af2baf07cc872e716f93e11694e868bb9adaabfa6b647: Status 404 returned error can't find the container with id 15d236f9584e5c4ea80af2baf07cc872e716f93e11694e868bb9adaabfa6b647 Oct 30 21:22:19.011905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:19.011871 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" event={"ID":"1a3e99ee-1a34-4531-99e4-5a7a083bce3b","Type":"ContainerStarted","Data":"15d236f9584e5c4ea80af2baf07cc872e716f93e11694e868bb9adaabfa6b647"} Oct 30 21:22:20.019306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.019272 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" event={"ID":"1a3e99ee-1a34-4531-99e4-5a7a083bce3b","Type":"ContainerStarted","Data":"8e1f4f637ed9d0a841f08be9718acd9f82b7754d8be51cdd01447834056a6cf6"} Oct 30 21:22:20.100083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.100055 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:20.135603 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.135529 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" podStartSLOduration=2.537800465 podStartE2EDuration="3.135509428s" totalImagesPullingTime="597.708963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:20.135147806 +0000 UTC m=+654.117848225" watchObservedRunningTime="2026-10-30 21:22:20.135509428 +0000 UTC m=+654.118209848" Oct 30 21:22:20.853138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.853099 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r"] Oct 30 21:22:20.856910 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.856886 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.957911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.957874 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:22:20.960766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.960733 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r"] Oct 30 21:22:20.966705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.966679 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.966820 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.966714 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.966820 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.966739 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.966964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.966836 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.966964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.966914 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wvf\" (UniqueName: \"kubernetes.io/projected/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-kube-api-access-l7wvf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:20.967065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:20.967048 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068223 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wvf\" (UniqueName: \"kubernetes.io/projected/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-kube-api-access-l7wvf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068288 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068346 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068369 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068390 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068420 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.068747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068726 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.069003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068791 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.069003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.068820 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.071408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.071388 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.071595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.071578 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.128572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.128493 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wvf\" (UniqueName: \"kubernetes.io/projected/258abee6-16f7-47a1-8ba8-0cc1b54c00ec-kube-api-access-l7wvf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dg85r\" (UID: \"258abee6-16f7-47a1-8ba8-0cc1b54c00ec\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.170488 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.170451 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:21.544956 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:21.544923 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r"] Oct 30 21:22:21.546457 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:21.546430 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258abee6_16f7_47a1_8ba8_0cc1b54c00ec.slice/crio-b9607ab27acd8cedfdf5d919316620335b2079099c9af11c9225f369267906be WatchSource:0}: Error finding container b9607ab27acd8cedfdf5d919316620335b2079099c9af11c9225f369267906be: Status 404 returned error can't find the container with id b9607ab27acd8cedfdf5d919316620335b2079099c9af11c9225f369267906be Oct 30 21:22:22.031000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:22.030965 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" event={"ID":"258abee6-16f7-47a1-8ba8-0cc1b54c00ec","Type":"ContainerStarted","Data":"b9607ab27acd8cedfdf5d919316620335b2079099c9af11c9225f369267906be"} Oct 30 21:22:23.039038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:23.039000 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" event={"ID":"258abee6-16f7-47a1-8ba8-0cc1b54c00ec","Type":"ContainerStarted","Data":"badce0cb7d41af3027a5dfe92269cd841bb9bd7c914499419fea7ff4ff9a4621"} Oct 30 21:22:23.156412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:23.156356 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" podStartSLOduration=2.572066632 podStartE2EDuration="3.156339626s" totalImagesPullingTime="584.272994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:23.154257605 +0000 UTC m=+657.136958025" watchObservedRunningTime="2026-10-30 21:22:23.156339626 +0000 UTC m=+657.139040081" Oct 30 21:22:23.170928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:23.170898 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:28.099724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:28.099680 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:28.112169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:28.112140 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:29.086583 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:29.086558 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-l7xtj" Oct 30 21:22:31.170805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:31.170767 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:31.183240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:31.183215 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:32.106259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:32.106227 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dg85r" Oct 30 21:22:37.541478 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:22:37.541390 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:22:48.623170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.623133 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:22:48.627065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.627038 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.731976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.731941 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:22:48.745736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745701 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.745922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745786 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.745922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745815 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.745922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745872 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.746044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745923 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.746044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745949 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.746044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.745972 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrl7m\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.746150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.746078 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.746150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.746097 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847328 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847272 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847343 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847456 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847503 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847569 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847596 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847620 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrl7m\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847692 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847711 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847765 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.847828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.847792 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.848036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.848014 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.848332 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.848294 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.848805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.848760 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.850804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.850777 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.851357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.851304 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.905038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.904959 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.905038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.905028 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrl7m\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\") pod \"maas-default-gateway-openshift-default-6cc458d887-78vsx\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:48.940920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:48.940883 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:49.376143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:49.376111 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:22:49.377284 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:49.377254 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda515bb63_8481_4694_b6ca_88f711f1e4c2.slice/crio-ef73e4183b4222f8d34c2a37c13288373089c2f59263b0e5e85683605a07a838 WatchSource:0}: Error finding container ef73e4183b4222f8d34c2a37c13288373089c2f59263b0e5e85683605a07a838: Status 404 returned error can't find the container with id ef73e4183b4222f8d34c2a37c13288373089c2f59263b0e5e85683605a07a838 Oct 30 21:22:49.379471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:49.379436 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:22:49.379552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:49.379511 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:22:49.379552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:49.379542 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:22:50.228916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:50.228878 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" event={"ID":"a515bb63-8481-4694-b6ca-88f711f1e4c2","Type":"ContainerStarted","Data":"4426f84e172d1595da6e268e09326046f6a7526dafad9a46394de2b1f03a822d"} Oct 30 21:22:50.229433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:50.228929 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" event={"ID":"a515bb63-8481-4694-b6ca-88f711f1e4c2","Type":"ContainerStarted","Data":"ef73e4183b4222f8d34c2a37c13288373089c2f59263b0e5e85683605a07a838"} Oct 30 21:22:50.941096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:50.941055 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:50.942556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:50.942532 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:50.942620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:50.942584 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:51.941956 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:51.941918 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:51.942482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:51.941989 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:52.646706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.646653 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podStartSLOduration=4.646638082 podStartE2EDuration="4.646638082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.445001304 +0000 UTC m=+684.427701723" watchObservedRunningTime="2026-10-30 21:22:52.646638082 +0000 UTC m=+686.629338505" Oct 30 21:22:52.647689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.647662 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-848f4b5756-qpp8k"] Oct 30 21:22:52.651651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.651631 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.735545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.735511 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-848f4b5756-qpp8k"] Oct 30 21:22:52.782051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.782020 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:22:52.782198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.782111 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvcbr\"" Oct 30 21:22:52.784757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.784732 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:22:52.786386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.786357 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmcfg\" (UniqueName: \"kubernetes.io/projected/e570d1ef-3aea-4199-8784-919aac381ad6-kube-api-access-dmcfg\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.786534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.786517 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570d1ef-3aea-4199-8784-919aac381ad6-tls-cert\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.788327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.788298 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:52.840071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.840041 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:22:52.840291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.840122 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:22:52.840291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.840264 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5c6fz\"" Oct 30 21:22:52.841333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.841302 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:22:52.887706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.887672 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570d1ef-3aea-4199-8784-919aac381ad6-tls-cert\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.887706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.887710 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78jl\" (UniqueName: \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:52.887883 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.887788 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmcfg\" (UniqueName: \"kubernetes.io/projected/e570d1ef-3aea-4199-8784-919aac381ad6-kube-api-access-dmcfg\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.887919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.887900 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:52.890080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.890058 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e570d1ef-3aea-4199-8784-919aac381ad6-tls-cert\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.941642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.941569 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:52.941642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.941625 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:52.944837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.944811 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmcfg\" (UniqueName: \"kubernetes.io/projected/e570d1ef-3aea-4199-8784-919aac381ad6-kube-api-access-dmcfg\") pod \"authorino-848f4b5756-qpp8k\" (UID: \"e570d1ef-3aea-4199-8784-919aac381ad6\") " pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.980357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.980305 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848f4b5756-qpp8k" Oct 30 21:22:52.988875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.988849 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:52.988994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.988908 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78jl\" (UniqueName: \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:52.991277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:52.991255 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:53.045272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.045243 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78jl\" (UniqueName: \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\") pod \"ai-gateway-controller-6cf7bcbfb6-7mwd6\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:53.134669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.134629 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:22:53.382477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.382450 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-848f4b5756-qpp8k"] Oct 30 21:22:53.384001 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:53.383973 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode570d1ef_3aea_4199_8784_919aac381ad6.slice/crio-f8440bbd04e7719f479562953fce990f3858c7f36bca03d389ccf2d93b078dd8 WatchSource:0}: Error finding container f8440bbd04e7719f479562953fce990f3858c7f36bca03d389ccf2d93b078dd8: Status 404 returned error can't find the container with id f8440bbd04e7719f479562953fce990f3858c7f36bca03d389ccf2d93b078dd8 Oct 30 21:22:53.526543 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.526514 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:22:53.527472 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:22:53.527445 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ec0a34_5532_49bd_9b37_b50fb12c1714.slice/crio-fe8d251de597d3aeb393139f03d194779d14f27c495aaeb0d8266c715ade2fad WatchSource:0}: Error finding container fe8d251de597d3aeb393139f03d194779d14f27c495aaeb0d8266c715ade2fad: Status 404 returned error can't find the container with id fe8d251de597d3aeb393139f03d194779d14f27c495aaeb0d8266c715ade2fad Oct 30 21:22:53.942175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.942141 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:53.942371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:53.942216 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:54.255476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:54.255442 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" event={"ID":"c5ec0a34-5532-49bd-9b37-b50fb12c1714","Type":"ContainerStarted","Data":"fe8d251de597d3aeb393139f03d194779d14f27c495aaeb0d8266c715ade2fad"} Oct 30 21:22:54.256795 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:54.256765 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-848f4b5756-qpp8k" event={"ID":"e570d1ef-3aea-4199-8784-919aac381ad6","Type":"ContainerStarted","Data":"f8440bbd04e7719f479562953fce990f3858c7f36bca03d389ccf2d93b078dd8"} Oct 30 21:22:54.941975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:54.941890 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:54.941975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:54.941947 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:55.266370 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:55.266325 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-848f4b5756-qpp8k" event={"ID":"e570d1ef-3aea-4199-8784-919aac381ad6","Type":"ContainerStarted","Data":"9deb5220952e238cae247423d56fa7454cbd1d3f2d88162169353fb6e6d0e973"} Oct 30 21:22:55.382663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:55.382607 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-848f4b5756-qpp8k" podStartSLOduration=2.133005237 podStartE2EDuration="3.382591811s" totalImagesPullingTime="1.249586574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:55.379781524 +0000 UTC m=+689.362481943" watchObservedRunningTime="2026-10-30 21:22:55.382591811 +0000 UTC m=+689.365292229" Oct 30 21:22:55.941602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:55.941559 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:55.941789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:55.941628 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:56.941830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:56.941798 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-78vsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:22:56.942463 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:56.941851 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 30 21:22:58.188766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:58.188721 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:58.189515 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:58.189493 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:22:58.190620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:22:58.190593 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:23:00.308932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:00.308899 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" event={"ID":"c5ec0a34-5532-49bd-9b37-b50fb12c1714","Type":"ContainerStarted","Data":"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32"} Oct 30 21:23:00.309362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:00.308964 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:23:00.425482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:00.425409 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" podStartSLOduration=2.274423882 podStartE2EDuration="8.425394078s" totalImagesPullingTime="6.150970196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:00.423562287 +0000 UTC m=+694.406262707" watchObservedRunningTime="2026-10-30 21:23:00.425394078 +0000 UTC m=+694.408094497" Oct 30 21:23:07.539237 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:23:07.539200 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:23:11.319645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:11.319614 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:23:26.468135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.468097 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:23:26.468655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.468295 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" podUID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" containerName="manager" containerID="cri-o://258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32" gracePeriod=30 Oct 30 21:23:26.726460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.726401 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:23:26.851629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.851597 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\" (UniqueName: \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\") pod \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " Oct 30 21:23:26.851843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.851814 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\") pod \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\" (UID: \"c5ec0a34-5532-49bd-9b37-b50fb12c1714\") " Oct 30 21:23:26.853897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.853864 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs" pod "c5ec0a34-5532-49bd-9b37-b50fb12c1714" (UID: "c5ec0a34-5532-49bd-9b37-b50fb12c1714"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:26.854005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.853895 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl" pod "c5ec0a34-5532-49bd-9b37-b50fb12c1714" (UID: "c5ec0a34-5532-49bd-9b37-b50fb12c1714"). InnerVolumeSpecName "kube-api-access-h78jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:26.952988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.952956 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ec0a34-5532-49bd-9b37-b50fb12c1714-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:26.952988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:26.952985 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78jl\" (UniqueName: \"kubernetes.io/projected/c5ec0a34-5532-49bd-9b37-b50fb12c1714-kube-api-access-h78jl\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:27.481911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.481876 4921 generic.go:356] "Generic (PLEG): container finished" podID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" containerID="258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32" exitCode=0 Oct 30 21:23:27.482373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.481919 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" Oct 30 21:23:27.482373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.481957 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" event={"ID":"c5ec0a34-5532-49bd-9b37-b50fb12c1714","Type":"ContainerDied","Data":"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32"} Oct 30 21:23:27.482373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.482011 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6" event={"ID":"c5ec0a34-5532-49bd-9b37-b50fb12c1714","Type":"ContainerDied","Data":"fe8d251de597d3aeb393139f03d194779d14f27c495aaeb0d8266c715ade2fad"} Oct 30 21:23:27.482373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.482033 4921 scope.go:122] "RemoveContainer" containerID="258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32" Oct 30 21:23:27.491961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.491942 4921 scope.go:122] "RemoveContainer" containerID="258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32" Oct 30 21:23:27.492217 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:23:27.492200 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32\": container with ID starting with 258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32 not found: ID does not exist" containerID="258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32" Oct 30 21:23:27.492253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.492226 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32"} err="failed to get container status \"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32\": rpc error: code = NotFound desc = could not find container \"258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32\": container with ID starting with 258fb146e9949402debddd99d5af28c729029a2f8e82254b98fb8a07a5eeef32 not found: ID does not exist" Oct 30 21:23:27.645231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.645195 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:23:27.647928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:27.647903 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-7mwd6"] Oct 30 21:23:28.167303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:28.167268 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" path="/var/lib/kubelet/pods/c5ec0a34-5532-49bd-9b37-b50fb12c1714/volumes" Oct 30 21:23:34.012867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.012828 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:23:34.013360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.013343 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" containerName="manager" Oct 30 21:23:34.013409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.013361 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" containerName="manager" Oct 30 21:23:34.013454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.013445 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ec0a34-5532-49bd-9b37-b50fb12c1714" containerName="manager" Oct 30 21:23:34.018127 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.018112 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.062450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.062392 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:23:34.066202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.066181 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.120219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120185 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120254 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120283 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7zj\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120307 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120459 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120499 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120530 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120620 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.120742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.120702 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.158024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.157994 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:23:34.221604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221575 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.221779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221623 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.221779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221644 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.221779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221766 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.221909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221804 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.221909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221867 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhr2\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.221909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221896 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.222060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221937 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.221997 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7zj\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222036 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222071 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.222189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222115 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.222189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222149 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222311 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222185 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222311 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222232 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222311 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222280 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222311 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222309 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222398 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.222560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222407 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222437 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222606 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222625 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.222770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.222708 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.224444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.224427 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.224815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.224795 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.239888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.239863 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:23:34.279740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.279671 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.279877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.279739 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7zj\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\") pod \"maas-default-gateway-openshift-default-6cc458d887-d8fzc\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.323890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.323853 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.323890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.323895 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.323946 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.323971 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324100 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324139 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324177 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324262 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhr2\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324290 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324378 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324488 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324525 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324793 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324774 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.324857 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.324777 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.326498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.326477 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.326594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.326507 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.331591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.331574 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:34.381352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.380882 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.381352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.381063 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhr2\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\") pod \"maas-default-gateway-openshift-default-6cc458d887-nmrsc\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.679474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.679370 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:34.763894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.763862 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:23:34.767471 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:23:34.767422 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e5f44b_0b34_4ca6_a254_f9342228db3e.slice/crio-f05458aa31a05d6cc9622941a807427bc890806eee568596ab50576129d3b91f WatchSource:0}: Error finding container f05458aa31a05d6cc9622941a807427bc890806eee568596ab50576129d3b91f: Status 404 returned error can't find the container with id f05458aa31a05d6cc9622941a807427bc890806eee568596ab50576129d3b91f Oct 30 21:23:34.770040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.770000 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:34.770108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.770091 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:34.770148 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:34.770136 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:35.065719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.065688 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:23:35.067617 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:23:35.067593 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b6645b_8bbe_4bad_9b93_639d63c02104.slice/crio-b5da7d0c1e4c61c2f2948a3ccc90689867bbf7ea73f49580ae6de0a522637826 WatchSource:0}: Error finding container b5da7d0c1e4c61c2f2948a3ccc90689867bbf7ea73f49580ae6de0a522637826: Status 404 returned error can't find the container with id b5da7d0c1e4c61c2f2948a3ccc90689867bbf7ea73f49580ae6de0a522637826 Oct 30 21:23:35.070116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.070087 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:35.070186 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.070150 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:35.070186 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.070177 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:23:35.536199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.536155 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" event={"ID":"f1b6645b-8bbe-4bad-9b93-639d63c02104","Type":"ContainerStarted","Data":"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc"} Oct 30 21:23:35.536443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.536226 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" event={"ID":"f1b6645b-8bbe-4bad-9b93-639d63c02104","Type":"ContainerStarted","Data":"b5da7d0c1e4c61c2f2948a3ccc90689867bbf7ea73f49580ae6de0a522637826"} Oct 30 21:23:35.538063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.538030 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" event={"ID":"b2e5f44b-0b34-4ca6-a254-f9342228db3e","Type":"ContainerStarted","Data":"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f"} Oct 30 21:23:35.538165 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.538087 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" event={"ID":"b2e5f44b-0b34-4ca6-a254-f9342228db3e","Type":"ContainerStarted","Data":"f05458aa31a05d6cc9622941a807427bc890806eee568596ab50576129d3b91f"} Oct 30 21:23:35.654305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.654248 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podStartSLOduration=2.6542336349999998 podStartE2EDuration="2.654233635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:35.65350238 +0000 UTC m=+729.636202799" watchObservedRunningTime="2026-10-30 21:23:35.654233635 +0000 UTC m=+729.636934051" Oct 30 21:23:35.680142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.680110 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:35.775013 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:35.774963 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podStartSLOduration=2.7749495680000003 podStartE2EDuration="2.774949568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:35.773771799 +0000 UTC m=+729.756472219" watchObservedRunningTime="2026-10-30 21:23:35.774949568 +0000 UTC m=+729.757650055" Oct 30 21:23:36.337342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:36.334065 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:36.337342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:36.334244 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:36.337342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:36.334291 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:36.681138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:36.681062 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:36.681368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:36.681343 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:37.342341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:37.341704 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:37.342341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:37.341765 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:37.571731 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:23:37.571683 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:23:37.679812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:37.679719 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:37.679812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:37.679786 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:38.332870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:38.332834 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:38.333189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:38.333145 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:38.680068 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:38.679972 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:38.680068 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:38.680033 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:39.333812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:39.333769 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:39.334103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:39.334073 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:39.680119 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:39.680025 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:39.680119 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:39.680081 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:40.333779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:40.333720 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:40.334063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:40.334032 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:40.680194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:40.680110 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:40.680194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:40.680165 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:41.332289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:41.332249 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:41.332497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:41.332336 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:41.680091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:41.680010 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:41.680091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:41.680067 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:42.332247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:42.332208 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-d8fzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:23:42.332442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:42.332272 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 21:23:42.680083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:42.679973 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nmrsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:23:42.680083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:42.680042 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 21:23:43.336761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.336726 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:43.337272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.337223 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:43.338090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.338069 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:23:43.684589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.684513 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:43.685033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.684797 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:23:43.685704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:23:43.685688 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:24:04.123147 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.123093 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:24:04.127353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.127333 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139047 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139081 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139098 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139271 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139300 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139348 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xl89\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139397 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139448 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.139481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.139473 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.201358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.201327 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:24:04.228175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.228149 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f52c3-openshift-default-dockercfg-ftmnt\"" Oct 30 21:24:04.240723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240690 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.240880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240723 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.240880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240742 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240900 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240932 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241004 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.240977 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xl89\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241006 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241078 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241105 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241441 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241640 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241472 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241556 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241654 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.241831 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.241694 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.243368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.243348 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.243589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.243570 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.297764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.297733 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.298116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.298095 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xl89\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.440833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.440745 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:04.764709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.764670 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:24:04.769654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.769631 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.824340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.824289 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-be9451-openshift-default-dockercfg-lwh6g\"" Oct 30 21:24:04.830523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.830493 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:24:04.849379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849334 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849396 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849501 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849576 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bld8q\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849683 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849619 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849697 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849743 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849769 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.849898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.849880 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.932912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.932881 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:24:04.934455 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:04.934428 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bb6d9b_27e0_4b71_be79_c1c66cba0fb9.slice/crio-d1f61b03ebfd256dbee88e0c1c3c225f951fdcd907339551c715ce706eb4b2d9 WatchSource:0}: Error finding container d1f61b03ebfd256dbee88e0c1c3c225f951fdcd907339551c715ce706eb4b2d9: Status 404 returned error can't find the container with id d1f61b03ebfd256dbee88e0c1c3c225f951fdcd907339551c715ce706eb4b2d9 Oct 30 21:24:04.936835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.936802 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:04.936929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.936869 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:04.936929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.936906 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:04.951230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951157 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951202 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951232 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951257 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951334 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951444 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951491 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951522 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951574 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bld8q\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951573 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951623 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951736 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.951969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.951948 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.952516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.952497 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.954951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.954928 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:04.955096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:04.955074 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:05.007533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.007505 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:05.007636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.007614 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bld8q\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:05.083704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.083552 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:05.407625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.407541 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:24:05.412671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.412647 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.464307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.464272 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8ce31a-openshift-default-dockercfg-kn28f\"" Oct 30 21:24:05.478377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.478283 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:24:05.522771 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.522748 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:24:05.524247 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:05.524216 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72fa16b_e51e_4a6d_8993_69b9033e8e77.slice/crio-77ffd041dabbce71b03f4e0c33ee19fc0b62223fc284b5b6506847d12931d19c WatchSource:0}: Error finding container 77ffd041dabbce71b03f4e0c33ee19fc0b62223fc284b5b6506847d12931d19c: Status 404 returned error can't find the container with id 77ffd041dabbce71b03f4e0c33ee19fc0b62223fc284b5b6506847d12931d19c Oct 30 21:24:05.526829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.526780 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:05.526909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.526867 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:05.526909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.526901 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.568831 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.568928 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.568976 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.569017 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.569035 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.569068 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.569127 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84mv\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.570261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.569175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.571292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.571079 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673126 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673260 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673336 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673374 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673416 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673497 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84mv\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673565 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673623 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673699 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.673981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.673716 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.674348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.674292 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.674348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.674327 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.674546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.674512 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.674546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.674532 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.676489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.676462 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.676619 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.676563 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.729383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.729347 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerStarted","Data":"55573edce635bf993fae5955fb4fef3d9d2ad91814bb3140f62e43f486d6bcfe"} Oct 30 21:24:05.729541 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.729411 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerStarted","Data":"d1f61b03ebfd256dbee88e0c1c3c225f951fdcd907339551c715ce706eb4b2d9"} Oct 30 21:24:05.730701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.730672 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.730822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.730792 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84mv\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:05.731227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.731206 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerStarted","Data":"326aeee22586cb838654c115728511bd908cd93b315022aeb1f855879ab592bf"} Oct 30 21:24:05.731279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.731246 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerStarted","Data":"77ffd041dabbce71b03f4e0c33ee19fc0b62223fc284b5b6506847d12931d19c"} Oct 30 21:24:05.847091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.846814 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podStartSLOduration=1.846797002 podStartE2EDuration="1.846797002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:05.846145585 +0000 UTC m=+759.828846005" watchObservedRunningTime="2026-10-30 21:24:05.846797002 +0000 UTC m=+759.829497423" Oct 30 21:24:05.964398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:05.964304 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podStartSLOduration=1.964284965 podStartE2EDuration="1.964284965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:05.962722965 +0000 UTC m=+759.945423387" watchObservedRunningTime="2026-10-30 21:24:05.964284965 +0000 UTC m=+759.946985386" Oct 30 21:24:06.026108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.026077 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:06.091486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.090474 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:06.107855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.107832 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:06.441860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.441775 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:06.446846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.446821 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:06.481939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.481902 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:24:06.483499 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:06.483467 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3c5391_56f1_4ce8_b8f6_3e6603ce27ac.slice/crio-79ead91c378d86d6d251e06227aede9392c17ffddbcf119460d7478dbc29fee6 WatchSource:0}: Error finding container 79ead91c378d86d6d251e06227aede9392c17ffddbcf119460d7478dbc29fee6: Status 404 returned error can't find the container with id 79ead91c378d86d6d251e06227aede9392c17ffddbcf119460d7478dbc29fee6 Oct 30 21:24:06.486155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.486119 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:06.486244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.486186 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:06.486244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.486230 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:06.740152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.740115 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerStarted","Data":"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf"} Oct 30 21:24:06.740420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.740178 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerStarted","Data":"79ead91c378d86d6d251e06227aede9392c17ffddbcf119460d7478dbc29fee6"} Oct 30 21:24:06.741357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.740652 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:06.741357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.740701 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:06.741771 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.741470 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:06.741771 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.741711 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:24:06.864644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:06.864514 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podStartSLOduration=1.8644944780000001 podStartE2EDuration="1.864494478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:06.862419028 +0000 UTC m=+760.845119447" watchObservedRunningTime="2026-10-30 21:24:06.864494478 +0000 UTC m=+760.847194898" Oct 30 21:24:07.027377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:07.027261 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:07.036566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:07.036540 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:07.540527 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:24:07.540491 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:24:07.746525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:07.746492 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:07.747339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:07.747278 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:18.645965 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.645926 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:24:18.650894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.650874 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.707682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.707652 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:24:18.743974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.743936 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2zd\" (UniqueName: \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.744305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.744285 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.752415 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.752387 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-0a4eab\"" Oct 30 21:24:18.752742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.752691 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-0a4eab-dockercfg-55mqq\"" Oct 30 21:24:18.804695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.804660 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:24:18.808571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.808553 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:18.845810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.845779 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.845974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.845878 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2zd\" (UniqueName: \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.846477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.846457 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.853663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.853633 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:24:18.901841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.901766 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2zd\" (UniqueName: \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\") pod \"payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:18.946411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.946379 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:18.946552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.946429 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mng6r\" (UniqueName: \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:18.964218 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:18.964190 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:19.047291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.047254 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mng6r\" (UniqueName: \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:19.047508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.047492 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:19.048162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.048145 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:19.104062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.103953 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mng6r\" (UniqueName: \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\") pod \"payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:19.121798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.121762 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:19.342674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.342650 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:24:19.344161 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:19.344139 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2115dda4_e1d6_47ad_a064_e961685640b1.slice/crio-44afe8ef80cfe52cf4b046213887bdf343bc58fd6e07f3f2a9db006b9dd12ae9 WatchSource:0}: Error finding container 44afe8ef80cfe52cf4b046213887bdf343bc58fd6e07f3f2a9db006b9dd12ae9: Status 404 returned error can't find the container with id 44afe8ef80cfe52cf4b046213887bdf343bc58fd6e07f3f2a9db006b9dd12ae9 Oct 30 21:24:19.502310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.502280 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:24:19.502913 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:19.502891 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13f9ac9_43d1_428a_b248_0d513323f20b.slice/crio-e5b65d0887fa65e386a094ed653f3098b6f7102db9f2ace65ac5a130747b9046 WatchSource:0}: Error finding container e5b65d0887fa65e386a094ed653f3098b6f7102db9f2ace65ac5a130747b9046: Status 404 returned error can't find the container with id e5b65d0887fa65e386a094ed653f3098b6f7102db9f2ace65ac5a130747b9046 Oct 30 21:24:19.819418 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.819334 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" event={"ID":"e13f9ac9-43d1-428a-b248-0d513323f20b","Type":"ContainerStarted","Data":"e5b65d0887fa65e386a094ed653f3098b6f7102db9f2ace65ac5a130747b9046"} Oct 30 21:24:19.820639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:19.820604 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" event={"ID":"2115dda4-e1d6-47ad-a064-e961685640b1","Type":"ContainerStarted","Data":"44afe8ef80cfe52cf4b046213887bdf343bc58fd6e07f3f2a9db006b9dd12ae9"} Oct 30 21:24:22.849587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:22.849492 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" event={"ID":"e13f9ac9-43d1-428a-b248-0d513323f20b","Type":"ContainerStarted","Data":"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4"} Oct 30 21:24:22.849587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:22.849585 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:22.851423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:22.851392 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" event={"ID":"2115dda4-e1d6-47ad-a064-e961685640b1","Type":"ContainerStarted","Data":"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798"} Oct 30 21:24:22.851581 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:22.851557 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:22.971060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:22.971002 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" podStartSLOduration=1.863205267 podStartE2EDuration="4.970989656s" totalImagesPullingTime="3.107784389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:22.970624802 +0000 UTC m=+776.953325221" watchObservedRunningTime="2026-10-30 21:24:22.970989656 +0000 UTC m=+776.953690074" Oct 30 21:24:23.089861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.089807 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" podStartSLOduration=1.932315303 podStartE2EDuration="5.089793511s" totalImagesPullingTime="3.157478208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:23.086899352 +0000 UTC m=+777.069599771" watchObservedRunningTime="2026-10-30 21:24:23.089793511 +0000 UTC m=+777.072493929" Oct 30 21:24:23.640808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.640773 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:24:23.644833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.644808 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.699612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.699563 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-38b832\"" Oct 30 21:24:23.702890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.702861 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5bz\" (UniqueName: \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.703053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.702927 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.703572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.703549 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:24:23.749627 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.749574 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-38b832-dockercfg-28dbm\"" Oct 30 21:24:23.804463 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.804428 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5bz\" (UniqueName: \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.804657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.804512 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.805217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.805181 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.863362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.863310 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5bz\" (UniqueName: \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\") pod \"payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:23.958460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:23.958420 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:24.337577 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:24.337520 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:24:24.338823 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:24.338792 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418cfe05_e0f9_42ff_bcc9_2aebc69e1f29.slice/crio-c8fdae2459bfa1c75357442ca59df56247ab5cf51ed9b9879132f22d02ad862f WatchSource:0}: Error finding container c8fdae2459bfa1c75357442ca59df56247ab5cf51ed9b9879132f22d02ad862f: Status 404 returned error can't find the container with id c8fdae2459bfa1c75357442ca59df56247ab5cf51ed9b9879132f22d02ad862f Oct 30 21:24:24.867595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:24.867558 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" event={"ID":"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29","Type":"ContainerStarted","Data":"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b"} Oct 30 21:24:24.867986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:24.867609 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" event={"ID":"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29","Type":"ContainerStarted","Data":"c8fdae2459bfa1c75357442ca59df56247ab5cf51ed9b9879132f22d02ad862f"} Oct 30 21:24:24.867986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:24.867669 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:24.980828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:24.980775 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" podStartSLOduration=1.980761298 podStartE2EDuration="1.980761298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:24.979580694 +0000 UTC m=+778.962281113" watchObservedRunningTime="2026-10-30 21:24:24.980761298 +0000 UTC m=+778.963461756" Oct 30 21:24:25.157921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.157842 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:24:25.165482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.164488 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.217932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.217899 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-38b832-serving-cert\"" Oct 30 21:24:25.218097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218054 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wqlw4\"" Oct 30 21:24:25.218280 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218257 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:24:25.218412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218371 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.218630 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218601 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.218739 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218697 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvzh\" (UniqueName: \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.218885 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.218774 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.221216 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.221189 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:24:25.319746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.319712 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.319919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.319805 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.319919 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:24:25.319853 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-serving-cert: secret "maas-api-e2e-worker-w2-38b832-serving-cert" not found Oct 30 21:24:25.319919 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:24:25.319911 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls podName:9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:25.819895706 +0000 UTC m=+779.802596103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls") pod "maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" (UID: "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33") : secret "maas-api-e2e-worker-w2-38b832-serving-cert" not found Oct 30 21:24:25.320038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.319856 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvzh\" (UniqueName: \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.320069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.320053 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.320515 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.320493 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.322535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.322514 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.377377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.377309 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvzh\" (UniqueName: \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.825180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.825136 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:25.827699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:25.827675 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:26.079547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:26.079448 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:26.455447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:26.455416 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:24:26.456459 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:26.456430 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b08a3fc_adb2_4c90_9f4e_d0cd91e5ee33.slice/crio-7a46cb96a57bc2fe72c079e17c35ef057e291968d7c3a63a4b07edfa0d80c664 WatchSource:0}: Error finding container 7a46cb96a57bc2fe72c079e17c35ef057e291968d7c3a63a4b07edfa0d80c664: Status 404 returned error can't find the container with id 7a46cb96a57bc2fe72c079e17c35ef057e291968d7c3a63a4b07edfa0d80c664 Oct 30 21:24:26.881435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:26.881347 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" event={"ID":"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33","Type":"ContainerStarted","Data":"7a46cb96a57bc2fe72c079e17c35ef057e291968d7c3a63a4b07edfa0d80c664"} Oct 30 21:24:28.563604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.562262 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:24:28.570104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.570067 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.608260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.608152 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:24:28.626142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.622372 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8ce31a\"" Oct 30 21:24:28.652380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.652347 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-8ce31a-dockercfg-m9s87\"" Oct 30 21:24:28.662046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.662010 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.662218 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.662166 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nmt\" (UniqueName: \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.763761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.763721 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.763969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.763798 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nmt\" (UniqueName: \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.764540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.764512 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.819914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.819881 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nmt\" (UniqueName: \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\") pod \"payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.889330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.889272 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:28.897705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.897671 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" event={"ID":"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33","Type":"ContainerStarted","Data":"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87"} Oct 30 21:24:28.898092 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:28.898065 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:29.016067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:29.015974 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" podStartSLOduration=2.029795464 podStartE2EDuration="4.015959912s" totalImagesPullingTime="1.986164448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:29.014733552 +0000 UTC m=+782.997433971" watchObservedRunningTime="2026-10-30 21:24:29.015959912 +0000 UTC m=+782.998660341" Oct 30 21:24:29.271975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:29.271946 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:24:29.273025 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:29.272999 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5351234_d2ba_40e9_87a0_6f3c3f04a1dd.slice/crio-5d442f1331902792855803bb9193c533de0c1be814eab670f9310ecdfb3fffd2 WatchSource:0}: Error finding container 5d442f1331902792855803bb9193c533de0c1be814eab670f9310ecdfb3fffd2: Status 404 returned error can't find the container with id 5d442f1331902792855803bb9193c533de0c1be814eab670f9310ecdfb3fffd2 Oct 30 21:24:29.952974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:29.952933 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" event={"ID":"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd","Type":"ContainerStarted","Data":"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c"} Oct 30 21:24:29.952974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:29.952979 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" event={"ID":"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd","Type":"ContainerStarted","Data":"5d442f1331902792855803bb9193c533de0c1be814eab670f9310ecdfb3fffd2"} Oct 30 21:24:29.953549 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:29.953013 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:30.078460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:30.078400 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" podStartSLOduration=2.078381817 podStartE2EDuration="2.078381817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:30.07746635 +0000 UTC m=+784.060166770" watchObservedRunningTime="2026-10-30 21:24:30.078381817 +0000 UTC m=+784.061082239" Oct 30 21:24:33.862921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:33.862893 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:24:33.863439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:33.862973 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:24:34.992394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:34.992364 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:24:35.876825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:35.876792 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:24:37.540538 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:24:37.540499 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:24:37.781481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:37.781439 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:24:37.781694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:37.781496 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:39.027218 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.027181 4921 generic.go:356] "Generic (PLEG): container finished" podID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerID="326aeee22586cb838654c115728511bd908cd93b315022aeb1f855879ab592bf" exitCode=0 Oct 30 21:24:39.027704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.027228 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerDied","Data":"326aeee22586cb838654c115728511bd908cd93b315022aeb1f855879ab592bf"} Oct 30 21:24:39.027704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.027629 4921 scope.go:122] "RemoveContainer" containerID="326aeee22586cb838654c115728511bd908cd93b315022aeb1f855879ab592bf" Oct 30 21:24:39.028899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.028869 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:39.028950 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.028940 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:39.028989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:39.028971 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:40.040627 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.040577 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerStarted","Data":"0e83ec614700c345a315b6b6f94e080e358027a6b2efe5fcc2ef8b3034ac3fb5"} Oct 30 21:24:40.091442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.088653 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:40.091442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.088773 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:40.091442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.088826 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:40.567628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.567360 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:24:40.572415 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.572390 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.628339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.625237 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f52c3-dockercfg-smfz4\"" Oct 30 21:24:40.643503 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.631062 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:24:40.643503 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.633436 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f52c3\"" Oct 30 21:24:40.648182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.648151 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.648337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.648271 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxvx\" (UniqueName: \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.749025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.748980 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.749199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.749115 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxvx\" (UniqueName: \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.749805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.749787 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.829891 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.829808 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxvx\" (UniqueName: \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\") pod \"payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:40.887710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:40.887675 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:41.088657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:41.087438 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:41.088657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:41.087493 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:41.353380 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:41.353282 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb7e337_dc8d_4827_a623_3e3eba0902fe.slice/crio-82cfa428ae2640c64b8bdedd88bef75ca672e3f4ab4af9957e892d218a3eb812 WatchSource:0}: Error finding container 82cfa428ae2640c64b8bdedd88bef75ca672e3f4ab4af9957e892d218a3eb812: Status 404 returned error can't find the container with id 82cfa428ae2640c64b8bdedd88bef75ca672e3f4ab4af9957e892d218a3eb812 Oct 30 21:24:41.391572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:41.388671 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:24:41.969943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:41.969916 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:24:42.077279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.077240 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" event={"ID":"3fb7e337-dc8d-4827-a623-3e3eba0902fe","Type":"ContainerStarted","Data":"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479"} Oct 30 21:24:42.077467 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.077305 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" event={"ID":"3fb7e337-dc8d-4827-a623-3e3eba0902fe","Type":"ContainerStarted","Data":"82cfa428ae2640c64b8bdedd88bef75ca672e3f4ab4af9957e892d218a3eb812"} Oct 30 21:24:42.077991 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.077970 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:42.084233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.084200 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:42.084410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.084252 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:42.263353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:42.263213 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" podStartSLOduration=2.26319304 podStartE2EDuration="2.26319304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:42.234728647 +0000 UTC m=+796.217429068" watchObservedRunningTime="2026-10-30 21:24:42.26319304 +0000 UTC m=+796.245893460" Oct 30 21:24:43.084143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:43.083983 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:43.084143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:43.084047 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:44.084693 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:44.084662 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:44.085605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:44.085576 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:45.084239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:45.084198 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:24:45.084470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:45.084259 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 21:24:46.088862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:46.088824 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:46.089372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:46.089348 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:46.091609 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:46.091580 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:24:51.175060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.175024 4921 generic.go:356] "Generic (PLEG): container finished" podID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerID="dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf" exitCode=0 Oct 30 21:24:51.175559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.175097 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerDied","Data":"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf"} Oct 30 21:24:51.175604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.175592 4921 scope.go:122] "RemoveContainer" containerID="dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf" Oct 30 21:24:51.176968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.176938 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:51.177098 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.177008 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:51.177098 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:51.177039 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:24:52.183659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:52.183627 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerStarted","Data":"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb"} Oct 30 21:24:52.746624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:52.746587 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:53.027200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:53.027108 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:53.028474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:53.028445 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:24:53.028637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:53.028497 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:24:54.027044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:54.027008 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:24:54.027548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:54.027067 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:24:54.096970 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:54.096935 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:24:55.027136 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:55.027094 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:24:55.027707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:55.027170 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:24:56.026968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:56.026930 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:24:56.027149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:56.026985 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:24:57.031362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.027622 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:24:57.031362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.027684 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 21:24:57.884063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.884027 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:24:57.887014 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.886998 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:57.990872 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.990831 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0a4eab\"/\"simulator-w5-0a4eab-kserve-self-signed-certs\"" Oct 30 21:24:57.990872 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.990855 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0a4eab\"/\"default-dockercfg-vkfnk\"" Oct 30 21:24:57.994146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:57.994121 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:24:58.040690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.039656 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041383 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwp4\" (UniqueName: \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041445 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041503 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041572 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041602 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.041653 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.042837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.042504 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0a4eab\"/\"kube-root-ca.crt\"" Oct 30 21:24:58.043432 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.043096 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0a4eab\"/\"openshift-service-ca.crt\"" Oct 30 21:24:58.044814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.044796 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:24:58.142659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142561 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.142659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142626 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.142659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142652 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.142971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142682 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.142971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142810 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwp4\" (UniqueName: \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.142971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.142849 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.143145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.143095 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.143196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.143142 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.143196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.143180 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.145095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.145068 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.145541 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.145519 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.199646 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.199617 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwp4\" (UniqueName: \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\") pod \"simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.498406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.498375 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:24:58.875843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:58.875815 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:24:58.877151 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:24:58.877129 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9669ef5_a505_49fd_95e4_45a59fe4780e.slice/crio-ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de WatchSource:0}: Error finding container ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de: Status 404 returned error can't find the container with id ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de Oct 30 21:24:59.258372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:59.258338 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" event={"ID":"e9669ef5-a505-49fd-95e4-45a59fe4780e","Type":"ContainerStarted","Data":"7b0e2da155eeeb417cd6a8abfd12774f4e294cda7d06c222740abeb1a19d9bea"} Oct 30 21:24:59.258765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:59.258384 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" event={"ID":"e9669ef5-a505-49fd-95e4-45a59fe4780e","Type":"ContainerStarted","Data":"ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de"} Oct 30 21:24:59.372863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:59.372808 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podStartSLOduration=2.3727945249999998 podStartE2EDuration="2.372794525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:59.372512811 +0000 UTC m=+813.355213231" watchObservedRunningTime="2026-10-30 21:24:59.372794525 +0000 UTC m=+813.355494944" Oct 30 21:24:59.499098 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:24:59.499057 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:25:06.500135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.500098 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:25:06.503846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.503820 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.541181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.541154 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.541376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.541251 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbk9\" (UniqueName: \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.541449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.541397 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.541449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.541422 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.548300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.548261 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:25:06.555133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.555111 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9f52c3-serving-cert\"" Oct 30 21:25:06.643173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.643123 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.643395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.643217 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbk9\" (UniqueName: \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.643395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.643297 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.643395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.643338 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.643395 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:25:06.643305 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-serving-cert: secret "maas-api-e2e-worker-w3-9f52c3-serving-cert" not found Oct 30 21:25:06.643614 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:25:06.643412 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls podName:fc8e7af9-57c0-4000-8d8c-5ee30662aaef nodeName:}" failed. No retries permitted until 2026-10-30 21:25:07.143390344 +0000 UTC m=+821.126090742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls") pod "maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" (UID: "fc8e7af9-57c0-4000-8d8c-5ee30662aaef") : secret "maas-api-e2e-worker-w3-9f52c3-serving-cert" not found Oct 30 21:25:06.644138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.644112 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.646815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.646793 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:06.701019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:06.700992 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbk9\" (UniqueName: \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:07.148276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:07.148239 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:07.150764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:07.150739 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:07.422659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:07.422528 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:07.635337 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:25:07.632690 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:25:07.811389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:07.811345 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:25:07.811812 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:25:07.811787 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8e7af9_57c0_4000_8d8c_5ee30662aaef.slice/crio-f54d0714e8cb1071c6bd00b9db374bda7c7ce48e452da9eb614c526343da8bcb WatchSource:0}: Error finding container f54d0714e8cb1071c6bd00b9db374bda7c7ce48e452da9eb614c526343da8bcb: Status 404 returned error can't find the container with id f54d0714e8cb1071c6bd00b9db374bda7c7ce48e452da9eb614c526343da8bcb Oct 30 21:25:08.318166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:08.318134 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" event={"ID":"fc8e7af9-57c0-4000-8d8c-5ee30662aaef","Type":"ContainerStarted","Data":"f54d0714e8cb1071c6bd00b9db374bda7c7ce48e452da9eb614c526343da8bcb"} Oct 30 21:25:08.499379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:08.499349 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:25:08.511423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:08.511398 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:25:09.328040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:09.328002 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" event={"ID":"fc8e7af9-57c0-4000-8d8c-5ee30662aaef","Type":"ContainerStarted","Data":"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd"} Oct 30 21:25:09.328464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:09.328220 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:09.339310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:09.339288 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:25:09.448179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:09.448129 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" podStartSLOduration=2.6052775710000002 podStartE2EDuration="3.448116739s" totalImagesPullingTime="842.839168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:09.444801648 +0000 UTC m=+823.427502068" watchObservedRunningTime="2026-10-30 21:25:09.448116739 +0000 UTC m=+823.430817165" Oct 30 21:25:15.351360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:15.351328 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:25:26.490664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.490575 4921 generic.go:356] "Generic (PLEG): container finished" podID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerID="55573edce635bf993fae5955fb4fef3d9d2ad91814bb3140f62e43f486d6bcfe" exitCode=0 Oct 30 21:25:26.490664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.490646 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerDied","Data":"55573edce635bf993fae5955fb4fef3d9d2ad91814bb3140f62e43f486d6bcfe"} Oct 30 21:25:26.491289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.491050 4921 scope.go:122] "RemoveContainer" containerID="55573edce635bf993fae5955fb4fef3d9d2ad91814bb3140f62e43f486d6bcfe" Oct 30 21:25:26.492389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.492358 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:26.492480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.492425 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:26.492480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:26.492457 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:25:27.524288 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:27.524240 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerStarted","Data":"f3ca8a9c6502568cbf62655ca97c46e9fea32707cceca28db9636aa3dc87baff"} Oct 30 21:25:28.441188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:28.441151 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:25:28.441504 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:28.441481 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:25:28.441564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:28.441533 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:25:29.441637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:29.441584 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:25:29.442151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:29.441650 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:25:30.441620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:30.441587 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:25:30.441825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:30.441645 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:25:31.441169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:31.441132 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:25:31.441393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:31.441195 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:25:32.441532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:32.441495 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:25:32.441920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:32.441550 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 21:25:33.445631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:33.445602 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:25:33.446025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:33.445914 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:25:33.446539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:25:33.446524 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:25:37.545521 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:25:37.545422 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:26:07.540967 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:26:07.540921 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:26:26.266890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:26:26.266859 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:26:26.268863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:26:26.268841 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:26:26.277829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:26:26.277810 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:26:26.279631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:26:26.279613 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:26:37.541675 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:26:37.541626 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:27:07.542018 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:27:07.541934 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:27:37.540822 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:27:37.540781 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:27:44.597206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.597149 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:27:44.601205 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.601185 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.670983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.670954 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:27:44.698257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698233 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.698392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698271 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.698392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698369 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.698472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698417 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8pm\" (UniqueName: \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.698541 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698527 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.698574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.698554 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.704185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.704161 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9ac5c5\"/\"openshift-service-ca.crt\"" Oct 30 21:27:44.704302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.704216 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9ac5c5\"/\"kube-root-ca.crt\"" Oct 30 21:27:44.704302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.704240 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9ac5c5\"/\"premium-w6-9ac5c5-kserve-self-signed-certs\"" Oct 30 21:27:44.704420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.704352 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9ac5c5\"/\"default-dockercfg-8tlqp\"" Oct 30 21:27:44.799651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799618 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.799817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799657 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.799817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799730 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.799817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799772 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.799974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799833 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.799974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.799882 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8pm\" (UniqueName: \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.800193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.800158 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.800282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.800194 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.800282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.800210 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.802031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.802006 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.802304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.802284 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.856568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.856514 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8pm\" (UniqueName: \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\") pod \"premium-w6-9ac5c5-kserve-ff547468b-nt875\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:44.913770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:44.913746 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:45.317173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:45.317145 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:27:45.317731 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:27:45.317705 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2089674d_33e5_406a_8f53_55ece03fe7ef.slice/crio-070b5219c599a86d69e99b8a33c5e1f3656459a678fa6ded0bc9e1d4e5ee543b WatchSource:0}: Error finding container 070b5219c599a86d69e99b8a33c5e1f3656459a678fa6ded0bc9e1d4e5ee543b: Status 404 returned error can't find the container with id 070b5219c599a86d69e99b8a33c5e1f3656459a678fa6ded0bc9e1d4e5ee543b Oct 30 21:27:45.556969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:45.556931 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" event={"ID":"2089674d-33e5-406a-8f53-55ece03fe7ef","Type":"ContainerStarted","Data":"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b"} Oct 30 21:27:45.556969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:45.556977 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" event={"ID":"2089674d-33e5-406a-8f53-55ece03fe7ef","Type":"ContainerStarted","Data":"070b5219c599a86d69e99b8a33c5e1f3656459a678fa6ded0bc9e1d4e5ee543b"} Oct 30 21:27:45.693718 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:45.693617 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podStartSLOduration=1.693602694 podStartE2EDuration="1.693602694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:45.691946864 +0000 UTC m=+979.674647281" watchObservedRunningTime="2026-10-30 21:27:45.693602694 +0000 UTC m=+979.676303110" Oct 30 21:27:45.914405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:45.914361 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:54.914121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:54.914084 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:54.926710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:54.926682 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:27:55.639375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:27:55.639346 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:28:05.788954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.788925 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:28:05.792924 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.792907 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.844903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.844874 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-847c62-openshift-default-dockercfg-67hhh\"" Oct 30 21:28:05.875055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875022 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875072 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875157 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875197 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875305 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875403 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875462 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnw6\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875519 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.875563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.875544 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.898459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.898428 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:28:05.976732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976699 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnw6\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.976938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976759 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.976938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976881 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.976938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976926 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976963 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.976998 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977025 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977090 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977205 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977542 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977565 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977832 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977872 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.977945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.977880 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.979548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.979529 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:05.979859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:05.979840 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:06.032881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.032854 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:06.033009 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.032884 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnw6\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:06.106909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.106834 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:06.532046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.532018 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:28:06.532932 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:28:06.532906 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78820725_24be_428c_a0e0_1a4ccf83f479.slice/crio-c6ecde31b7ae0723bdc4aaffdedd5e96d33e13318c1ce54555c1f960bf267d60 WatchSource:0}: Error finding container c6ecde31b7ae0723bdc4aaffdedd5e96d33e13318c1ce54555c1f960bf267d60: Status 404 returned error can't find the container with id c6ecde31b7ae0723bdc4aaffdedd5e96d33e13318c1ce54555c1f960bf267d60 Oct 30 21:28:06.539601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.539545 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:06.539732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.539650 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:06.539732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.539712 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:06.706124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.706090 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" event={"ID":"78820725-24be-428c-a0e0-1a4ccf83f479","Type":"ContainerStarted","Data":"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06"} Oct 30 21:28:06.706304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.706143 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" event={"ID":"78820725-24be-428c-a0e0-1a4ccf83f479","Type":"ContainerStarted","Data":"c6ecde31b7ae0723bdc4aaffdedd5e96d33e13318c1ce54555c1f960bf267d60"} Oct 30 21:28:06.829395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:06.829267 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podStartSLOduration=1.829247396 podStartE2EDuration="1.829247396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:06.823737122 +0000 UTC m=+1000.806437543" watchObservedRunningTime="2026-10-30 21:28:06.829247396 +0000 UTC m=+1000.811947816" Oct 30 21:28:07.084734 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.084662 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:28:07.088586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.088567 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.124458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.109517 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:07.124458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.123650 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:07.124458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.123703 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:07.140181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.140152 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-38b832-openshift-default-dockercfg-q7nbg\"" Oct 30 21:28:07.149472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.149440 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:28:07.191849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.191817 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.191979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.191921 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.191979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.191948 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.191979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.191969 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.192080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.192062 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.192121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.192106 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9cl\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.192245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.192229 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.192290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.192277 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.192383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.192369 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293411 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293448 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293519 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293557 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293577 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293598 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293625 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.293705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293659 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9cl\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293728 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293880 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293951 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.293975 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.294050 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.294530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.294510 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.295916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.295889 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.296199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.296183 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.350221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.350148 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.350392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.350281 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9cl\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.419392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.411169 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:07.555825 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:07.555789 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:28:07.852571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.852545 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:28:07.853898 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:28:07.853872 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2257752d_2b24_4829_af38_5e4e8f02f3b9.slice/crio-1c26865b33dede4445e7c0dc88f401e91e56fe957be27de02cc32b0fc895512f WatchSource:0}: Error finding container 1c26865b33dede4445e7c0dc88f401e91e56fe957be27de02cc32b0fc895512f: Status 404 returned error can't find the container with id 1c26865b33dede4445e7c0dc88f401e91e56fe957be27de02cc32b0fc895512f Oct 30 21:28:07.856209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.856176 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:07.856292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.856279 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:07.856351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:07.856327 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:28:08.107996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:08.107564 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:08.107996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:08.107618 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:08.763336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:08.762637 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" event={"ID":"2257752d-2b24-4829-af38-5e4e8f02f3b9","Type":"ContainerStarted","Data":"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f"} Oct 30 21:28:08.763336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:08.762692 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" event={"ID":"2257752d-2b24-4829-af38-5e4e8f02f3b9","Type":"ContainerStarted","Data":"1c26865b33dede4445e7c0dc88f401e91e56fe957be27de02cc32b0fc895512f"} Oct 30 21:28:09.115159 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:09.114665 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:09.115159 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:09.114727 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:09.421745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:09.421655 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:09.427484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:09.427453 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:28:09.427739 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:09.427697 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:28:10.109779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:10.109736 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:10.110575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:10.110541 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:10.424391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:10.423700 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:28:10.424391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:10.423792 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:28:11.123649 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:11.123615 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:11.123921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:11.123891 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:11.423399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:11.422717 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:28:11.423399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:11.422782 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:28:12.111697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:12.111662 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:28:12.112291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:12.112261 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:28:12.423516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:12.423418 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:28:12.423768 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:12.423738 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:28:13.231616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.231585 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:13.233185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.233158 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:13.234611 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.234590 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:28:13.351220 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.351154 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podStartSLOduration=6.351137921 podStartE2EDuration="6.351137921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:09.024554895 +0000 UTC m=+1003.007255314" watchObservedRunningTime="2026-10-30 21:28:13.351137921 +0000 UTC m=+1007.333838339" Oct 30 21:28:13.423348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.422726 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:28:13.423348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:13.422803 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:28:14.112590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.112558 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9"] Oct 30 21:28:14.116500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.116481 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.175308 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.175276 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9"] Oct 30 21:28:14.219007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.218976 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:28:14.219215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.219095 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:28:14.243183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.243148 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.243673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.243223 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.243673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.243360 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r49vn\" (UniqueName: \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.344665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.344620 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.344865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.344718 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r49vn\" (UniqueName: \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.344865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.344788 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.345263 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.345241 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.347145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.347120 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.411487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.411407 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r49vn\" (UniqueName: \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\") pod \"maas-api-revoke-sub-1f6c35d7-cgqk9\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.440072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.438860 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:14.483931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.483906 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:14.485021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.484990 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:14.486366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.486347 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:28:14.848103 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:28:14.848061 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6cb422_0d2e_4500_ad58_11f0548b1513.slice/crio-1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7 WatchSource:0}: Error finding container 1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7: Status 404 returned error can't find the container with id 1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7 Oct 30 21:28:14.848596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.848570 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9"] Oct 30 21:28:14.850737 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:14.850714 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:28:15.838132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:15.838101 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" event={"ID":"cf6cb422-0d2e-4500-ad58-11f0548b1513","Type":"ContainerStarted","Data":"1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7"} Oct 30 21:28:18.197097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.197064 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:28:18.206677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.206648 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.261427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.261375 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:28:18.291185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291149 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.291185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291192 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.291429 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291409 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.291477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291457 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.291589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291568 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8s6\" (UniqueName: \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.291675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.291615 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.305610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.305581 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f52c3\"/\"openshift-service-ca.crt\"" Oct 30 21:28:18.305610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.305593 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f52c3\"/\"kube-root-ca.crt\"" Oct 30 21:28:18.305825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.305633 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f52c3\"/\"default-dockercfg-fmsr2\"" Oct 30 21:28:18.307833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.307781 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f52c3\"/\"premium-w3-9f52c3-kserve-self-signed-certs\"" Oct 30 21:28:18.392554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392525 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.392765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392577 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8s6\" (UniqueName: \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.392765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392616 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.392765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392675 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.392765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392700 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.393000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392857 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.393000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.392971 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.393227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.393196 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.393227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.393219 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.394980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.394945 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.395745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.395724 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.452350 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.452289 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8s6\" (UniqueName: \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\") pod \"premium-w3-9f52c3-kserve-68dd9888f-cppbv\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.522706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.521943 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:18.961461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:18.961433 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:28:18.962190 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:28:18.962166 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd572a994_8a3a_425f_ac9a_51bb2a5016db.slice/crio-8654035f11d053b0da052f5d62759fe23cf1f04e5942dd816cf3f568eb59ac0d WatchSource:0}: Error finding container 8654035f11d053b0da052f5d62759fe23cf1f04e5942dd816cf3f568eb59ac0d: Status 404 returned error can't find the container with id 8654035f11d053b0da052f5d62759fe23cf1f04e5942dd816cf3f568eb59ac0d Oct 30 21:28:19.867038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:19.866951 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" event={"ID":"d572a994-8a3a-425f-ac9a-51bb2a5016db","Type":"ContainerStarted","Data":"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f"} Oct 30 21:28:19.867038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:19.866995 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" event={"ID":"d572a994-8a3a-425f-ac9a-51bb2a5016db","Type":"ContainerStarted","Data":"8654035f11d053b0da052f5d62759fe23cf1f04e5942dd816cf3f568eb59ac0d"} Oct 30 21:28:19.869232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:19.869201 4921 generic.go:356] "Generic (PLEG): container finished" podID="cf6cb422-0d2e-4500-ad58-11f0548b1513" containerID="3a58663587fa09e7438b1c57648f0e817d849cc5c57210361f1b65c1aba9f7ba" exitCode=0 Oct 30 21:28:19.869410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:19.869284 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" event={"ID":"cf6cb422-0d2e-4500-ad58-11f0548b1513","Type":"ContainerDied","Data":"3a58663587fa09e7438b1c57648f0e817d849cc5c57210361f1b65c1aba9f7ba"} Oct 30 21:28:19.981006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:19.980955 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podStartSLOduration=1.980937823 podStartE2EDuration="1.980937823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:19.979953338 +0000 UTC m=+1013.962653756" watchObservedRunningTime="2026-10-30 21:28:19.980937823 +0000 UTC m=+1013.963638259" Oct 30 21:28:20.522581 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:20.522549 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:21.022743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.022719 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:21.125524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.125495 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\" (UniqueName: \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\") pod \"cf6cb422-0d2e-4500-ad58-11f0548b1513\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " Oct 30 21:28:21.125695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.125657 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\") pod \"cf6cb422-0d2e-4500-ad58-11f0548b1513\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " Oct 30 21:28:21.125742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.125723 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\") pod \"cf6cb422-0d2e-4500-ad58-11f0548b1513\" (UID: \"cf6cb422-0d2e-4500-ad58-11f0548b1513\") " Oct 30 21:28:21.126111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.126054 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca" pod "cf6cb422-0d2e-4500-ad58-11f0548b1513" (UID: "cf6cb422-0d2e-4500-ad58-11f0548b1513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:21.127624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.127600 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca" pod "cf6cb422-0d2e-4500-ad58-11f0548b1513" (UID: "cf6cb422-0d2e-4500-ad58-11f0548b1513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:21.127680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.127669 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn" pod "cf6cb422-0d2e-4500-ad58-11f0548b1513" (UID: "cf6cb422-0d2e-4500-ad58-11f0548b1513"). InnerVolumeSpecName "kube-api-access-r49vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:21.227521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.227493 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:21.227521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.227518 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r49vn\" (UniqueName: \"kubernetes.io/projected/cf6cb422-0d2e-4500-ad58-11f0548b1513-kube-api-access-r49vn\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:21.227521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.227528 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6cb422-0d2e-4500-ad58-11f0548b1513-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:21.883353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.883302 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" Oct 30 21:28:21.883494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.883343 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9" event={"ID":"cf6cb422-0d2e-4500-ad58-11f0548b1513","Type":"ContainerDied","Data":"1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7"} Oct 30 21:28:21.883494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:21.883424 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1edbd6023dd0f057083e3359db832f31e35a54c240095a82a1ab68bbfb00e7c7" Oct 30 21:28:28.522394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:28.522361 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:28.534708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:28.534679 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:28.938859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:28.938778 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:28:36.370043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.369999 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:28:36.370558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.370295 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" containerID="cri-o://4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb" gracePeriod=30 Oct 30 21:28:36.470646 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.470603 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:28:36.470912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.470882 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" containerID="cri-o://4426f84e172d1595da6e268e09326046f6a7526dafad9a46394de2b1f03a822d" gracePeriod=30 Oct 30 21:28:36.475335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.474516 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:28:36.475335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.474731 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" containerID="cri-o://02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc" gracePeriod=30 Oct 30 21:28:36.479327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.479281 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:28:36.479568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:36.479526 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" containerID="cri-o://21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f" gracePeriod=30 Oct 30 21:28:37.541912 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:37.541812 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:28:40.167428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:40.167390 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-ggsrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:28:40.167834 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:40.167456 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:28:41.701469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.700072 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:28:41.774098 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.774065 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:28:41.805010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.804979 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:28:41.827194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827153 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827225 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827251 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827306 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827595 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.827700 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827665 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827829 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827913 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827872 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.827968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827935 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.828016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.827990 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\") pod \"5af6193a-70ed-409d-9fed-7f7e41d590cb\" (UID: \"5af6193a-70ed-409d-9fed-7f7e41d590cb\") " Oct 30 21:28:41.828755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.828501 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.828755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.828635 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.828755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.828723 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.829012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.828957 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:41.829231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.829206 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.830856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.830827 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "kube-api-access-sxs99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.830955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.830919 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.831017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.830960 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.835722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.835399 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo" pod "5af6193a-70ed-409d-9fed-7f7e41d590cb" (UID: "5af6193a-70ed-409d-9fed-7f7e41d590cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:41.847310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.847277 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc"] Oct 30 21:28:41.847992 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.847968 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" Oct 30 21:28:41.848078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.847993 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" Oct 30 21:28:41.848078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848012 4921 cpu_manager.go:414] "containerMap: removing container" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848019 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848041 4921 cpu_manager.go:414] "containerMap: removing container" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848049 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848082 4921 cpu_manager.go:414] "containerMap: removing container" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" Oct 30 21:28:41.848238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848089 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" Oct 30 21:28:41.848238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848103 4921 cpu_manager.go:414] "containerMap: removing container" podUID="cf6cb422-0d2e-4500-ad58-11f0548b1513" containerName="revoke-keys" Oct 30 21:28:41.848238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848110 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6cb422-0d2e-4500-ad58-11f0548b1513" containerName="revoke-keys" Oct 30 21:28:41.848238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848234 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerName="istio-proxy" Oct 30 21:28:41.848392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848246 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848258 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerName="istio-proxy" Oct 30 21:28:41.848392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848272 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerName="istio-proxy" Oct 30 21:28:41.848392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.848283 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6cb422-0d2e-4500-ad58-11f0548b1513" containerName="revoke-keys" Oct 30 21:28:41.852556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.852537 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:41.910329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.910246 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc"] Oct 30 21:28:41.916435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.916334 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:28:41.916698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.916678 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:28:41.928922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.928887 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.929158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929137 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.929372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929355 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.929575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929558 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.929734 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929719 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929875 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929914 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929959 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929999 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930034 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930067 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.930106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930103 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930136 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930184 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930252 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930399 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930471 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\") pod \"f1b6645b-8bbe-4bad-9b93-639d63c02104\" (UID: \"f1b6645b-8bbe-4bad-9b93-639d63c02104\") " Oct 30 21:28:41.930652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.930525 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\") pod \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\" (UID: \"b2e5f44b-0b34-4ca6-a254-f9342228db3e\") " Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931070 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931094 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxs99\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-kube-api-access-sxs99\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931109 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931122 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931137 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931151 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931166 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af6193a-70ed-409d-9fed-7f7e41d590cb-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931181 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af6193a-70ed-409d-9fed-7f7e41d590cb-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.929799 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931098 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931274 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931532 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931623 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.931689 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.932036 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.932262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.932078 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.933444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.933402 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.933694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.933650 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.933843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.933814 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.934229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.934205 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:41.934554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.934532 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.934642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.934569 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "kube-api-access-pb7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.936372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.935920 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo" pod "b2e5f44b-0b34-4ca6-a254-f9342228db3e" (UID: "b2e5f44b-0b34-4ca6-a254-f9342228db3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:41.936372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.936054 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.936645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.936598 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.936971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:41.936949 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2" pod "f1b6645b-8bbe-4bad-9b93-639d63c02104" (UID: "f1b6645b-8bbe-4bad-9b93-639d63c02104"). InnerVolumeSpecName "kube-api-access-bnhr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.032496 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.032578 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.032704 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6fk\" (UniqueName: \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.032980 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.032998 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033013 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7zj\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-kube-api-access-pb7zj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033027 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033039 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033053 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033065 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1b6645b-8bbe-4bad-9b93-639d63c02104-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033078 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhr2\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-kube-api-access-bnhr2\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033092 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033105 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033118 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033132 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033145 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033157 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033170 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033184 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e5f44b-0b34-4ca6-a254-f9342228db3e-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033197 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e5f44b-0b34-4ca6-a254-f9342228db3e-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.034379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.033223 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1b6645b-8bbe-4bad-9b93-639d63c02104-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.039145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.037900 4921 generic.go:356] "Generic (PLEG): container finished" podID="5af6193a-70ed-409d-9fed-7f7e41d590cb" containerID="4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb" exitCode=0 Oct 30 21:28:42.039145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.038018 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerDied","Data":"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb"} Oct 30 21:28:42.039145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.038062 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" event={"ID":"5af6193a-70ed-409d-9fed-7f7e41d590cb","Type":"ContainerDied","Data":"d3fc506a6a4b6004b529e7f94f28a042e397287996473c714767c2b7d3a0b430"} Oct 30 21:28:42.039145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.038084 4921 scope.go:122] "RemoveContainer" containerID="4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb" Oct 30 21:28:42.039145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.038206 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf" Oct 30 21:28:42.043170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.043140 4921 generic.go:356] "Generic (PLEG): container finished" podID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerID="4426f84e172d1595da6e268e09326046f6a7526dafad9a46394de2b1f03a822d" exitCode=0 Oct 30 21:28:42.043377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.043354 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" event={"ID":"a515bb63-8481-4694-b6ca-88f711f1e4c2","Type":"ContainerDied","Data":"4426f84e172d1595da6e268e09326046f6a7526dafad9a46394de2b1f03a822d"} Oct 30 21:28:42.045194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.045168 4921 generic.go:356] "Generic (PLEG): container finished" podID="f1b6645b-8bbe-4bad-9b93-639d63c02104" containerID="02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc" exitCode=0 Oct 30 21:28:42.045290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.045261 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" event={"ID":"f1b6645b-8bbe-4bad-9b93-639d63c02104","Type":"ContainerDied","Data":"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc"} Oct 30 21:28:42.045377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.045302 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" event={"ID":"f1b6645b-8bbe-4bad-9b93-639d63c02104","Type":"ContainerDied","Data":"b5da7d0c1e4c61c2f2948a3ccc90689867bbf7ea73f49580ae6de0a522637826"} Oct 30 21:28:42.045430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.045409 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc" Oct 30 21:28:42.050445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.050418 4921 generic.go:356] "Generic (PLEG): container finished" podID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" containerID="21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f" exitCode=0 Oct 30 21:28:42.050574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.050465 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" event={"ID":"b2e5f44b-0b34-4ca6-a254-f9342228db3e","Type":"ContainerDied","Data":"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f"} Oct 30 21:28:42.050574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.050507 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" event={"ID":"b2e5f44b-0b34-4ca6-a254-f9342228db3e","Type":"ContainerDied","Data":"f05458aa31a05d6cc9622941a807427bc890806eee568596ab50576129d3b91f"} Oct 30 21:28:42.053116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.053099 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc" Oct 30 21:28:42.063280 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.063246 4921 scope.go:122] "RemoveContainer" containerID="4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005" Oct 30 21:28:42.079751 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.079730 4921 scope.go:122] "RemoveContainer" containerID="4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb" Oct 30 21:28:42.080776 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:42.080305 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb\": container with ID starting with 4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb not found: ID does not exist" containerID="4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb" Oct 30 21:28:42.080776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.080363 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb"} err="failed to get container status \"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb\": rpc error: code = NotFound desc = could not find container \"4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb\": container with ID starting with 4f015453c461e1c9d8791bd4132e5ad4517699a06a95a3391f24664df701dbcb not found: ID does not exist" Oct 30 21:28:42.080776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.080382 4921 scope.go:122] "RemoveContainer" containerID="4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005" Oct 30 21:28:42.081529 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:42.081498 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005\": container with ID starting with 4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005 not found: ID does not exist" containerID="4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005" Oct 30 21:28:42.081636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.081558 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005"} err="failed to get container status \"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005\": rpc error: code = NotFound desc = could not find container \"4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005\": container with ID starting with 4bc85eb7674c8f6091b3f52d7d03b3c8c4636c3f8579e52c9c7522b97875d005 not found: ID does not exist" Oct 30 21:28:42.081636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.081584 4921 scope.go:122] "RemoveContainer" containerID="02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc" Oct 30 21:28:42.093965 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.093946 4921 scope.go:122] "RemoveContainer" containerID="02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc" Oct 30 21:28:42.094282 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:42.094258 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc\": container with ID starting with 02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc not found: ID does not exist" containerID="02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc" Oct 30 21:28:42.094379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.094293 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc"} err="failed to get container status \"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc\": rpc error: code = NotFound desc = could not find container \"02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc\": container with ID starting with 02f0ba26f1db6a0d9874611231ace0b204f5e572af4ac64c073a1ba79d2a91cc not found: ID does not exist" Oct 30 21:28:42.094379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.094341 4921 scope.go:122] "RemoveContainer" containerID="21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f" Oct 30 21:28:42.136735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.135491 4921 scope.go:122] "RemoveContainer" containerID="21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f" Oct 30 21:28:42.137132 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:28:42.137067 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f\": container with ID starting with 21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f not found: ID does not exist" containerID="21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f" Oct 30 21:28:42.137132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.137097 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f"} err="failed to get container status \"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f\": rpc error: code = NotFound desc = could not find container \"21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f\": container with ID starting with 21671ca0b5fc95ccfaaa0898b33050e9c25e3ff9ec0eb0ac4b4ebfa5c936b73f not found: ID does not exist" Oct 30 21:28:42.137878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.137854 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.137989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.137954 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.138062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.138012 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6fk\" (UniqueName: \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.138871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.138850 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.146402 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.146381 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.163107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.163077 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:28:42.194562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.194486 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6fk\" (UniqueName: \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\") pod \"maas-api-revoke-sub-5881bb19-x99nc\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.341999 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.341967 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342016 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342055 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342161 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342194 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342234 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342258 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342325 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342365 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\") pod \"a515bb63-8481-4694-b6ca-88f711f1e4c2\" (UID: \"a515bb63-8481-4694-b6ca-88f711f1e4c2\") " Oct 30 21:28:42.342661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342401 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:42.342661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342425 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:42.342661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342600 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:42.342803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342699 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:42.342869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342793 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:42.342869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342815 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.342869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342832 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.342869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342846 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.342869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.342859 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a515bb63-8481-4694-b6ca-88f711f1e4c2-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.344609 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.344584 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:42.344718 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.344588 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:42.344780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.344765 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:42.345074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.345058 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m" pod "a515bb63-8481-4694-b6ca-88f711f1e4c2" (UID: "a515bb63-8481-4694-b6ca-88f711f1e4c2"). InnerVolumeSpecName "kube-api-access-rrl7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:42.356138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.356118 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:28:42.359395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.359373 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-ggsrf"] Oct 30 21:28:42.443457 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.443425 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.443457 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.443453 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.443663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.443467 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a515bb63-8481-4694-b6ca-88f711f1e4c2-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.443663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.443480 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrl7m\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-kube-api-access-rrl7m\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.443663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.443493 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a515bb63-8481-4694-b6ca-88f711f1e4c2-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:42.468340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.467433 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:42.471575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.471548 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:28:42.476451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.476306 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-d8fzc"] Oct 30 21:28:42.584464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.584435 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:28:42.594829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.594799 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nmrsc"] Oct 30 21:28:42.849566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:42.849529 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc"] Oct 30 21:28:42.851008 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:28:42.850985 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254724c_69c0_4fad_a626_4365e9b75103.slice/crio-d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4 WatchSource:0}: Error finding container d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4: Status 404 returned error can't find the container with id d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4 Oct 30 21:28:43.061662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.061612 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" event={"ID":"9254724c-69c0-4fad-a626-4365e9b75103","Type":"ContainerStarted","Data":"d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4"} Oct 30 21:28:43.073367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.073336 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" Oct 30 21:28:43.073500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.073339 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx" event={"ID":"a515bb63-8481-4694-b6ca-88f711f1e4c2","Type":"ContainerDied","Data":"ef73e4183b4222f8d34c2a37c13288373089c2f59263b0e5e85683605a07a838"} Oct 30 21:28:43.073500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.073461 4921 scope.go:122] "RemoveContainer" containerID="4426f84e172d1595da6e268e09326046f6a7526dafad9a46394de2b1f03a822d" Oct 30 21:28:43.237113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.237085 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:28:43.240816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:43.240793 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-78vsx"] Oct 30 21:28:44.082062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.082022 4921 generic.go:356] "Generic (PLEG): container finished" podID="9254724c-69c0-4fad-a626-4365e9b75103" containerID="f868fd8bd71f02169b48616bdf7c5ae87ed91b0ce65706cfb3f69c2ce2d337f9" exitCode=0 Oct 30 21:28:44.082477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.082090 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" event={"ID":"9254724c-69c0-4fad-a626-4365e9b75103","Type":"ContainerDied","Data":"f868fd8bd71f02169b48616bdf7c5ae87ed91b0ce65706cfb3f69c2ce2d337f9"} Oct 30 21:28:44.168483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.168453 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af6193a-70ed-409d-9fed-7f7e41d590cb" path="/var/lib/kubelet/pods/5af6193a-70ed-409d-9fed-7f7e41d590cb/volumes" Oct 30 21:28:44.168919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.168904 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" path="/var/lib/kubelet/pods/a515bb63-8481-4694-b6ca-88f711f1e4c2/volumes" Oct 30 21:28:44.169440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.169419 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e5f44b-0b34-4ca6-a254-f9342228db3e" path="/var/lib/kubelet/pods/b2e5f44b-0b34-4ca6-a254-f9342228db3e/volumes" Oct 30 21:28:44.170197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:44.170177 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b6645b-8bbe-4bad-9b93-639d63c02104" path="/var/lib/kubelet/pods/f1b6645b-8bbe-4bad-9b93-639d63c02104/volumes" Oct 30 21:28:45.220706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.220682 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:45.277043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.277013 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\" (UniqueName: \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\") pod \"9254724c-69c0-4fad-a626-4365e9b75103\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " Oct 30 21:28:45.277169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.277052 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\") pod \"9254724c-69c0-4fad-a626-4365e9b75103\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " Oct 30 21:28:45.277169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.277101 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\") pod \"9254724c-69c0-4fad-a626-4365e9b75103\" (UID: \"9254724c-69c0-4fad-a626-4365e9b75103\") " Oct 30 21:28:45.277540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.277519 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca" pod "9254724c-69c0-4fad-a626-4365e9b75103" (UID: "9254724c-69c0-4fad-a626-4365e9b75103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:45.278996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.278973 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca" pod "9254724c-69c0-4fad-a626-4365e9b75103" (UID: "9254724c-69c0-4fad-a626-4365e9b75103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:28:45.279063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.278979 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk" pod "9254724c-69c0-4fad-a626-4365e9b75103" (UID: "9254724c-69c0-4fad-a626-4365e9b75103"). InnerVolumeSpecName "kube-api-access-qv6fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:45.378681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.378622 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:45.378681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.378643 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv6fk\" (UniqueName: \"kubernetes.io/projected/9254724c-69c0-4fad-a626-4365e9b75103-kube-api-access-qv6fk\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:45.378681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:45.378653 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9254724c-69c0-4fad-a626-4365e9b75103-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:28:46.095664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:46.095633 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" Oct 30 21:28:46.095844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:46.095670 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc" event={"ID":"9254724c-69c0-4fad-a626-4365e9b75103","Type":"ContainerDied","Data":"d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4"} Oct 30 21:28:46.095844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:28:46.095715 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4beff42f7d5b5e92563b179d595289cd3c5afb19644e65f40b501627116fab4" Oct 30 21:29:01.098719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.098690 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:29:01.099392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099371 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9254724c-69c0-4fad-a626-4365e9b75103" containerName="revoke-keys" Oct 30 21:29:01.099536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099395 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9254724c-69c0-4fad-a626-4365e9b75103" containerName="revoke-keys" Oct 30 21:29:01.099536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099457 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" Oct 30 21:29:01.099536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099465 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" Oct 30 21:29:01.099654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099609 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a515bb63-8481-4694-b6ca-88f711f1e4c2" containerName="istio-proxy" Oct 30 21:29:01.099654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.099624 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9254724c-69c0-4fad-a626-4365e9b75103" containerName="revoke-keys" Oct 30 21:29:01.107008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.106988 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.163713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.163682 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9ac5c5-openshift-default-dockercfg-bnx6p\"" Oct 30 21:29:01.218420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.218374 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:29:01.242866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.242835 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.242877 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.242905 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ks8\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243013 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243053 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243107 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243206 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243270 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243251 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.243360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.243336 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344440 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344512 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344552 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344582 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344713 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344756 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344797 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ks8\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344825 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344849 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.344973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.344854 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.345145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.345119 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.345286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.345261 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.345391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.345371 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.345483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.345466 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.347146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.347120 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.347375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.347355 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.402590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.402521 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.402869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.402845 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ks8\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.421650 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.421626 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:01.854968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.854935 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:29:01.859536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.859497 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:01.859659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.859582 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:01.859659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:01.859627 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:02.224115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.221453 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" event={"ID":"48a2f12e-567e-4b4f-aeef-84b45c0668f3","Type":"ContainerStarted","Data":"5dddfd2b00467acaad9f8eeb2ce7bf73bfd609b957a55db6eaff776fb5dbd98f"} Oct 30 21:29:02.224115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.221525 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" event={"ID":"48a2f12e-567e-4b4f-aeef-84b45c0668f3","Type":"ContainerStarted","Data":"3f764e02e1e50fed74b8909753fbe82c7e0b9bfabf75b3caf1b83cf61201f81a"} Oct 30 21:29:02.346087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.346024 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podStartSLOduration=1.346005202 podStartE2EDuration="1.346005202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:02.341370401 +0000 UTC m=+1056.324070819" watchObservedRunningTime="2026-10-30 21:29:02.346005202 +0000 UTC m=+1056.328705625" Oct 30 21:29:02.422918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.422872 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:02.424977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.424945 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:02.425130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:02.425005 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:03.423798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:03.423759 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:03.424227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:03.423839 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:04.423848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:04.423801 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:04.424387 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:04.423864 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:05.423855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:05.423807 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:05.424398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:05.423874 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:06.423341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:06.422922 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:06.423341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:06.422988 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:07.426339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:07.423729 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:07.426339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:07.423789 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:07.641335 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:29:07.637509 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:29:08.422867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:08.422826 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:29:08.423025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:08.422922 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:29:09.426369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:09.426309 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:09.426835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:09.426820 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:09.427608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:09.427588 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:29:13.887690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:13.887659 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7"] Oct 30 21:29:13.891673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:13.891653 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:13.943679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:13.943653 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:29:13.978867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:13.978835 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:29:13.996684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:13.996657 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7"] Oct 30 21:29:14.036521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.036488 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.036825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.036798 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95ph\" (UniqueName: \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.036928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.036852 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.137838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.137755 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.137986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.137971 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.138032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.138020 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95ph\" (UniqueName: \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.138569 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.138548 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.140589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.140563 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.194244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.194218 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95ph\" (UniqueName: \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\") pod \"maas-api-revoke-sub-6b1d6130-pr4j7\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.206089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.206064 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:14.577855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:14.577819 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7"] Oct 30 21:29:14.578375 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:29:14.578349 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086dd2f7_1cf3_45bc_88fc_ecc447df458d.slice/crio-8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16 WatchSource:0}: Error finding container 8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16: Status 404 returned error can't find the container with id 8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16 Oct 30 21:29:15.327408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.327371 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" event={"ID":"086dd2f7-1cf3-45bc-88fc-ecc447df458d","Type":"ContainerStarted","Data":"cb9b992318f0cc46a71a545997389f4c79da29359d532ce939c69a247e908e9f"} Oct 30 21:29:15.327808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.327426 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" event={"ID":"086dd2f7-1cf3-45bc-88fc-ecc447df458d","Type":"ContainerStarted","Data":"8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16"} Oct 30 21:29:15.441914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.441852 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" podStartSLOduration=2.441834028 podStartE2EDuration="2.441834028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:15.440291269 +0000 UTC m=+1069.422991681" watchObservedRunningTime="2026-10-30 21:29:15.441834028 +0000 UTC m=+1069.424534448" Oct 30 21:29:15.743876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.743842 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:29:15.748636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.748607 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.801277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.801243 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8ce31a\"/\"openshift-service-ca.crt\"" Oct 30 21:29:15.801453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.801411 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8ce31a\"/\"distinct-w1-8ce31a-kserve-self-signed-certs\"" Oct 30 21:29:15.801536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.801520 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8ce31a\"/\"default-dockercfg-zprht\"" Oct 30 21:29:15.802342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.802296 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8ce31a\"/\"kube-root-ca.crt\"" Oct 30 21:29:15.804764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.804740 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:29:15.858274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858237 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h685z\" (UniqueName: \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.858442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858393 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.858442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858425 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.858558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858537 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.858605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858594 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.858645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.858614 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960247 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h685z\" (UniqueName: \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960348 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960372 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960398 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960434 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960448 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960787 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960857 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960801 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.960892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.960865 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.962663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.962642 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:15.962942 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:15.962924 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:16.016897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:16.016823 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h685z\" (UniqueName: \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\") pod \"distinct-w1-8ce31a-kserve-689846b5d7-wdvll\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:16.060459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:16.060435 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:16.337782 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:16.337690 4921 generic.go:356] "Generic (PLEG): container finished" podID="086dd2f7-1cf3-45bc-88fc-ecc447df458d" containerID="cb9b992318f0cc46a71a545997389f4c79da29359d532ce939c69a247e908e9f" exitCode=0 Oct 30 21:29:16.337782 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:16.337765 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" event={"ID":"086dd2f7-1cf3-45bc-88fc-ecc447df458d","Type":"ContainerDied","Data":"cb9b992318f0cc46a71a545997389f4c79da29359d532ce939c69a247e908e9f"} Oct 30 21:29:16.500344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:16.500239 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:29:17.349060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.349028 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" event={"ID":"b8936abe-5b55-4cea-a509-7399e237916d","Type":"ContainerStarted","Data":"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367"} Oct 30 21:29:17.349396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.349076 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" event={"ID":"b8936abe-5b55-4cea-a509-7399e237916d","Type":"ContainerStarted","Data":"22cc89b0f0d68e30113709cb96755dd0d4fe499710b148192aacd43076cc98f2"} Oct 30 21:29:17.499421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.499397 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:17.580866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.580833 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\" (UniqueName: \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\") pod \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " Oct 30 21:29:17.581087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.581068 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\") pod \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " Oct 30 21:29:17.581181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.581126 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\") pod \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\" (UID: \"086dd2f7-1cf3-45bc-88fc-ecc447df458d\") " Oct 30 21:29:17.581419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.581395 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca" pod "086dd2f7-1cf3-45bc-88fc-ecc447df458d" (UID: "086dd2f7-1cf3-45bc-88fc-ecc447df458d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:17.581544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.581527 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:17.582955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.582935 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph" pod "086dd2f7-1cf3-45bc-88fc-ecc447df458d" (UID: "086dd2f7-1cf3-45bc-88fc-ecc447df458d"). InnerVolumeSpecName "kube-api-access-n95ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:17.583042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.583021 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca" pod "086dd2f7-1cf3-45bc-88fc-ecc447df458d" (UID: "086dd2f7-1cf3-45bc-88fc-ecc447df458d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:17.602594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.602507 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podStartSLOduration=2.602493244 podStartE2EDuration="2.602493244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:17.59890907 +0000 UTC m=+1071.581609488" watchObservedRunningTime="2026-10-30 21:29:17.602493244 +0000 UTC m=+1071.585193661" Oct 30 21:29:17.682408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.682377 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/086dd2f7-1cf3-45bc-88fc-ecc447df458d-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:17.682408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:17.682403 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95ph\" (UniqueName: \"kubernetes.io/projected/086dd2f7-1cf3-45bc-88fc-ecc447df458d-kube-api-access-n95ph\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:18.061154 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:18.061113 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:18.357901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:18.357815 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" Oct 30 21:29:18.357901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:18.357850 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7" event={"ID":"086dd2f7-1cf3-45bc-88fc-ecc447df458d","Type":"ContainerDied","Data":"8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16"} Oct 30 21:29:18.358489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:18.357903 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b425ffa43218f26dbc9bb715d1b2e5ba792a73d2df9272a1362f79eaf33ce16" Oct 30 21:29:25.816657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.816616 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk"] Oct 30 21:29:25.817150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.817137 4921 cpu_manager.go:414] "containerMap: removing container" podUID="086dd2f7-1cf3-45bc-88fc-ecc447df458d" containerName="revoke-keys" Oct 30 21:29:25.817204 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.817151 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086dd2f7-1cf3-45bc-88fc-ecc447df458d" containerName="revoke-keys" Oct 30 21:29:25.817247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.817238 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="086dd2f7-1cf3-45bc-88fc-ecc447df458d" containerName="revoke-keys" Oct 30 21:29:25.820773 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.820755 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.864096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.864067 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk"] Oct 30 21:29:25.889554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.889520 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.889693 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.889645 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljd6\" (UniqueName: \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.889842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.889821 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.909411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.909387 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:29:25.909540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.909481 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:29:25.991057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.991025 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.991203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.991107 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.991203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.991162 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljd6\" (UniqueName: \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.991743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.991720 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:25.993520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:25.993499 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:26.050934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.050907 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljd6\" (UniqueName: \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\") pod \"maas-api-revoke-sub-5ac356aa-jxrqk\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:26.061522 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.061501 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:26.073790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.073745 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:26.184130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.184110 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:29:26.193375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.193351 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:26.445507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.445475 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:29:26.568601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:26.568570 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk"] Oct 30 21:29:26.569812 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:29:26.569783 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode117bd3a_59b7_4798_9308_4f39c683cff6.slice/crio-a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf WatchSource:0}: Error finding container a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf: Status 404 returned error can't find the container with id a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf Oct 30 21:29:27.440805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:27.440767 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" event={"ID":"e117bd3a-59b7-4798-9308-4f39c683cff6","Type":"ContainerStarted","Data":"8fcc7c9677c0d4eae5696e68761df5928f994afc9a47da0ce3ce0caf9a50f3f3"} Oct 30 21:29:27.441261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:27.440829 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" event={"ID":"e117bd3a-59b7-4798-9308-4f39c683cff6","Type":"ContainerStarted","Data":"a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf"} Oct 30 21:29:27.554535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:27.554458 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" podStartSLOduration=2.554440182 podStartE2EDuration="2.554440182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:27.55393373 +0000 UTC m=+1081.536634148" watchObservedRunningTime="2026-10-30 21:29:27.554440182 +0000 UTC m=+1081.537140601" Oct 30 21:29:28.448032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:28.447994 4921 generic.go:356] "Generic (PLEG): container finished" podID="e117bd3a-59b7-4798-9308-4f39c683cff6" containerID="8fcc7c9677c0d4eae5696e68761df5928f994afc9a47da0ce3ce0caf9a50f3f3" exitCode=0 Oct 30 21:29:28.448431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:28.448076 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" event={"ID":"e117bd3a-59b7-4798-9308-4f39c683cff6","Type":"ContainerDied","Data":"8fcc7c9677c0d4eae5696e68761df5928f994afc9a47da0ce3ce0caf9a50f3f3"} Oct 30 21:29:29.612053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.612020 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:29.732433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.732354 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\" (UniqueName: \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\") pod \"e117bd3a-59b7-4798-9308-4f39c683cff6\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " Oct 30 21:29:29.732433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.732412 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\") pod \"e117bd3a-59b7-4798-9308-4f39c683cff6\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " Oct 30 21:29:29.732653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.732505 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\") pod \"e117bd3a-59b7-4798-9308-4f39c683cff6\" (UID: \"e117bd3a-59b7-4798-9308-4f39c683cff6\") " Oct 30 21:29:29.732794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.732759 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca" pod "e117bd3a-59b7-4798-9308-4f39c683cff6" (UID: "e117bd3a-59b7-4798-9308-4f39c683cff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:29.733008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.732985 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:29.734549 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.734520 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6" pod "e117bd3a-59b7-4798-9308-4f39c683cff6" (UID: "e117bd3a-59b7-4798-9308-4f39c683cff6"). InnerVolumeSpecName "kube-api-access-qljd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:29.734549 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.734534 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca" pod "e117bd3a-59b7-4798-9308-4f39c683cff6" (UID: "e117bd3a-59b7-4798-9308-4f39c683cff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:29.834222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.834192 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljd6\" (UniqueName: \"kubernetes.io/projected/e117bd3a-59b7-4798-9308-4f39c683cff6-kube-api-access-qljd6\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:29.834222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:29.834218 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e117bd3a-59b7-4798-9308-4f39c683cff6-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:30.462754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:30.462728 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" Oct 30 21:29:30.462921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:30.462766 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk" event={"ID":"e117bd3a-59b7-4798-9308-4f39c683cff6","Type":"ContainerDied","Data":"a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf"} Oct 30 21:29:30.462921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:30.462817 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1a4efa7207c2a207e58d357f6d20197fed63887414815782be7d81ddcce7daf" Oct 30 21:29:37.540825 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:29:37.540785 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:29:39.560418 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.560382 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5"] Oct 30 21:29:39.561838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.561811 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e117bd3a-59b7-4798-9308-4f39c683cff6" containerName="revoke-keys" Oct 30 21:29:39.561987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.561971 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e117bd3a-59b7-4798-9308-4f39c683cff6" containerName="revoke-keys" Oct 30 21:29:39.562229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.562212 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e117bd3a-59b7-4798-9308-4f39c683cff6" containerName="revoke-keys" Oct 30 21:29:39.567373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.567351 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.642339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.636094 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5"] Oct 30 21:29:39.671617 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.671585 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:29:39.672590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.672568 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:29:39.764606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.764569 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.765010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.764987 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsxp\" (UniqueName: \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.765226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.765206 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.875038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.874928 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.875379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.875353 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsxp\" (UniqueName: \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.875529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.875513 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.878128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.878095 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.892112 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.892079 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:39.955307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:39.955262 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsxp\" (UniqueName: \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\") pod \"maas-api-revoke-sub-a2b63bb1-k5tn5\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:40.190278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:40.190190 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:40.888411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:40.888370 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5"] Oct 30 21:29:40.890148 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:29:40.890117 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9475243_aeb8_4c2b_8fb8_cf4dab7167da.slice/crio-278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e WatchSource:0}: Error finding container 278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e: Status 404 returned error can't find the container with id 278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e Oct 30 21:29:41.606960 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:41.606867 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" event={"ID":"b9475243-aeb8-4c2b-8fb8-cf4dab7167da","Type":"ContainerStarted","Data":"0a5e033805254fac998da65b91a3b53086f4464257a49d3ebcf3497e556b71a2"} Oct 30 21:29:41.607266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:41.607228 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" event={"ID":"b9475243-aeb8-4c2b-8fb8-cf4dab7167da","Type":"ContainerStarted","Data":"278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e"} Oct 30 21:29:41.721894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:41.721834 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" podStartSLOduration=2.72181763 podStartE2EDuration="2.72181763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:41.721311095 +0000 UTC m=+1095.704011511" watchObservedRunningTime="2026-10-30 21:29:41.72181763 +0000 UTC m=+1095.704518050" Oct 30 21:29:42.615100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:42.615065 4921 generic.go:356] "Generic (PLEG): container finished" podID="b9475243-aeb8-4c2b-8fb8-cf4dab7167da" containerID="0a5e033805254fac998da65b91a3b53086f4464257a49d3ebcf3497e556b71a2" exitCode=0 Oct 30 21:29:42.615506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:42.615130 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" event={"ID":"b9475243-aeb8-4c2b-8fb8-cf4dab7167da","Type":"ContainerDied","Data":"0a5e033805254fac998da65b91a3b53086f4464257a49d3ebcf3497e556b71a2"} Oct 30 21:29:43.787905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.787881 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:43.926307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.926212 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\") pod \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " Oct 30 21:29:43.926498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.926437 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\") pod \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " Oct 30 21:29:43.926498 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.926472 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\" (UniqueName: \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\") pod \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\" (UID: \"b9475243-aeb8-4c2b-8fb8-cf4dab7167da\") " Oct 30 21:29:43.926864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.926831 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca" pod "b9475243-aeb8-4c2b-8fb8-cf4dab7167da" (UID: "b9475243-aeb8-4c2b-8fb8-cf4dab7167da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:43.926994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.926980 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:43.930689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.930664 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca" pod "b9475243-aeb8-4c2b-8fb8-cf4dab7167da" (UID: "b9475243-aeb8-4c2b-8fb8-cf4dab7167da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:43.941588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:43.941541 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp" pod "b9475243-aeb8-4c2b-8fb8-cf4dab7167da" (UID: "b9475243-aeb8-4c2b-8fb8-cf4dab7167da"). InnerVolumeSpecName "kube-api-access-2fsxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:44.028240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:44.028204 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fsxp\" (UniqueName: \"kubernetes.io/projected/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-kube-api-access-2fsxp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.028240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:44.028236 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9475243-aeb8-4c2b-8fb8-cf4dab7167da-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:29:44.631896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:44.631864 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" Oct 30 21:29:44.632087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:44.631900 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5" event={"ID":"b9475243-aeb8-4c2b-8fb8-cf4dab7167da","Type":"ContainerDied","Data":"278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e"} Oct 30 21:29:44.632087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:44.631962 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278fb6634b68cf734e96d6fa45e0b7f6b5f45fcfecc5edb6f759d11b5c94970e" Oct 30 21:29:52.204762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.204730 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:29:52.205242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.205230 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b9475243-aeb8-4c2b-8fb8-cf4dab7167da" containerName="revoke-keys" Oct 30 21:29:52.205299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.205243 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9475243-aeb8-4c2b-8fb8-cf4dab7167da" containerName="revoke-keys" Oct 30 21:29:52.205372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.205360 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9475243-aeb8-4c2b-8fb8-cf4dab7167da" containerName="revoke-keys" Oct 30 21:29:52.209836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.209818 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.265173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.265145 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0a4eab-openshift-default-dockercfg-f7j2g\"" Oct 30 21:29:52.314983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.314945 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315019 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9q7v\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315226 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315254 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315280 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315460 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315494 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.315530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.315518 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.318360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.318339 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:29:52.416417 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416391 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416422 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416439 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416491 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416514 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416536 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416592 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416653 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416721 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9q7v\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.416969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.416938 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.417128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.417100 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.417128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.417125 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.417293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.417178 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.417676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.417655 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.418695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.418673 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.419271 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.419246 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.472583 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.472532 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.472693 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.472676 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9q7v\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.523264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.523244 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:52.949134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.949097 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:29:52.949534 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:29:52.949505 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701aacc1_673e_4302_9bc3_a60d8fdd4405.slice/crio-ebcfe32ac44d7977d5d11c2751743485af3435a0b725a54e51f1b06fd47c7aec WatchSource:0}: Error finding container ebcfe32ac44d7977d5d11c2751743485af3435a0b725a54e51f1b06fd47c7aec: Status 404 returned error can't find the container with id ebcfe32ac44d7977d5d11c2751743485af3435a0b725a54e51f1b06fd47c7aec Oct 30 21:29:52.951745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.951712 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:52.951814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.951776 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:52.951814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:52.951802 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:29:53.692669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:53.692638 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" event={"ID":"701aacc1-673e-4302-9bc3-a60d8fdd4405","Type":"ContainerStarted","Data":"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c"} Oct 30 21:29:53.693056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:53.692686 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" event={"ID":"701aacc1-673e-4302-9bc3-a60d8fdd4405","Type":"ContainerStarted","Data":"ebcfe32ac44d7977d5d11c2751743485af3435a0b725a54e51f1b06fd47c7aec"} Oct 30 21:29:53.917337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:53.914757 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podStartSLOduration=1.9147348499999999 podStartE2EDuration="1.91473485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:53.858907074 +0000 UTC m=+1107.841607491" watchObservedRunningTime="2026-10-30 21:29:53.91473485 +0000 UTC m=+1107.897435269" Oct 30 21:29:54.525342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:54.525000 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:54.527376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:54.526554 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:29:54.527376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:54.526610 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:29:55.523627 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:55.523593 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:29:55.524010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:55.523642 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:29:56.523701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:56.523660 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:29:56.524191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:56.523741 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:29:57.523654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:57.523615 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:29:57.523840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:57.523690 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:29:58.529628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:58.529594 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:29:58.530219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:58.530176 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:29:59.528869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:59.528837 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:59.529350 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:59.529330 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:29:59.529984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:29:59.529960 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:30:00.168401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.168371 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:00.172505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.172481 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.174440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.174417 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:00.177992 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.177975 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:00.178088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.178072 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.181674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.181657 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.191469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.191450 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:00.195371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.195355 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.214215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214192 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.214348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214282 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68x5n\" (UniqueName: \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.214348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214339 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.214445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214356 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.214445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214389 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sj69\" (UniqueName: \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.214445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.214425 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhws\" (UniqueName: \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.224361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.224337 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:30:00.271419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.271396 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:00.274905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.274885 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:00.315045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315022 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.315184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315074 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.315184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315103 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbhz5\" (UniqueName: \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.315294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315228 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68x5n\" (UniqueName: \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.315294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315280 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.315471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315301 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.315471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315394 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sj69\" (UniqueName: \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.315471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315454 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhws\" (UniqueName: \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.315889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.315835 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.316109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.316090 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.316238 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.316221 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.326525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.326497 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:00.326635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.326561 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:00.374985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.374957 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhws\" (UniqueName: \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\") pod \"maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.375083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.375046 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68x5n\" (UniqueName: \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.375127 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.375109 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sj69\" (UniqueName: \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\") pod \"maas-api-key-cleanup-29889930-chcv7\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.416735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.416711 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.416848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.416745 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbhz5\" (UniqueName: \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.417304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.417284 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.474817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.474794 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbhz5\" (UniqueName: \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.487823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.487797 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:00.494910 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.494893 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:00.504749 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.504727 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:00.538568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.538534 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:00.901135 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:00.901092 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50311055_1eec_4dbb_bde6_b8e3ac556a82.slice/crio-dc232fc2797ce6f40de84598a60912a487e1245b61fb19cc07a037279d6d9b8c WatchSource:0}: Error finding container dc232fc2797ce6f40de84598a60912a487e1245b61fb19cc07a037279d6d9b8c: Status 404 returned error can't find the container with id dc232fc2797ce6f40de84598a60912a487e1245b61fb19cc07a037279d6d9b8c Oct 30 21:30:00.904746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.904719 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:00.951527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.951505 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:00.953166 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:00.953137 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c82d16c_6e6f_43fe_9379_473b42fbe6b4.slice/crio-fa37b5723c9fb64e0e7418d75f0e650febb3b8620065e6750bae54800fe3fb19 WatchSource:0}: Error finding container fa37b5723c9fb64e0e7418d75f0e650febb3b8620065e6750bae54800fe3fb19: Status 404 returned error can't find the container with id fa37b5723c9fb64e0e7418d75f0e650febb3b8620065e6750bae54800fe3fb19 Oct 30 21:30:00.997739 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:00.997703 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724383d8_40ac_4dae_bcce_8e73ede1cc6d.slice/crio-daeb87cc792f77e74893b9cbce53a8ac7968f01f8bf9136d103b0e418bf9a4cb WatchSource:0}: Error finding container daeb87cc792f77e74893b9cbce53a8ac7968f01f8bf9136d103b0e418bf9a4cb: Status 404 returned error can't find the container with id daeb87cc792f77e74893b9cbce53a8ac7968f01f8bf9136d103b0e418bf9a4cb Oct 30 21:30:00.997863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:00.997813 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:00.999967 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:00.999929 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd90a2a_30e0_4951_9f0f_dbe14230846f.slice/crio-77368c095d3f3a0db2dce290ece1a2c4b85ca4153a5e1836ee10ac18a43d9433 WatchSource:0}: Error finding container 77368c095d3f3a0db2dce290ece1a2c4b85ca4153a5e1836ee10ac18a43d9433: Status 404 returned error can't find the container with id 77368c095d3f3a0db2dce290ece1a2c4b85ca4153a5e1836ee10ac18a43d9433 Oct 30 21:30:01.004446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.003814 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:01.776829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.776796 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerStarted","Data":"0208ee8ce1eff583dc50d8520995ca030f108bfb2a64601191094b9464a17644"} Oct 30 21:30:01.777274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.776857 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerStarted","Data":"daeb87cc792f77e74893b9cbce53a8ac7968f01f8bf9136d103b0e418bf9a4cb"} Oct 30 21:30:01.778411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.778386 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerStarted","Data":"cb02cc159d1fec8ab7c725e56f5824a3c652579413b5b866233020182a11eb48"} Oct 30 21:30:01.778521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.778432 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerStarted","Data":"fa37b5723c9fb64e0e7418d75f0e650febb3b8620065e6750bae54800fe3fb19"} Oct 30 21:30:01.779950 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.779927 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerStarted","Data":"11e256982860913a3bf00884f6f06fe321d919347f5216cc2338473c983d632d"} Oct 30 21:30:01.780045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.779967 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerStarted","Data":"dc232fc2797ce6f40de84598a60912a487e1245b61fb19cc07a037279d6d9b8c"} Oct 30 21:30:01.781429 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.781404 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerStarted","Data":"da21b592bcfa552985654e187dcd2e70f9edc9f10dece31e436543ee7e541627"} Oct 30 21:30:01.781511 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.781452 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerStarted","Data":"77368c095d3f3a0db2dce290ece1a2c4b85ca4153a5e1836ee10ac18a43d9433"} Oct 30 21:30:01.892639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:01.892589 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" podStartSLOduration=1.892576147 podStartE2EDuration="1.892576147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.890134105 +0000 UTC m=+1115.872834523" watchObservedRunningTime="2026-10-30 21:30:01.892576147 +0000 UTC m=+1115.875276583" Oct 30 21:30:02.006307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.006265 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" podStartSLOduration=2.006252695 podStartE2EDuration="2.006252695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.002855953 +0000 UTC m=+1115.985556371" watchObservedRunningTime="2026-10-30 21:30:02.006252695 +0000 UTC m=+1115.988953135" Oct 30 21:30:02.118790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.118741 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" podStartSLOduration=2.118728811 podStartE2EDuration="2.118728811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.11642891 +0000 UTC m=+1116.099129371" watchObservedRunningTime="2026-10-30 21:30:02.118728811 +0000 UTC m=+1116.101429229" Oct 30 21:30:02.230656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.230606 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" podStartSLOduration=2.230590125 podStartE2EDuration="2.230590125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.229332466 +0000 UTC m=+1116.212032883" watchObservedRunningTime="2026-10-30 21:30:02.230590125 +0000 UTC m=+1116.213290544" Oct 30 21:30:02.790111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.790080 4921 generic.go:356] "Generic (PLEG): container finished" podID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerID="0208ee8ce1eff583dc50d8520995ca030f108bfb2a64601191094b9464a17644" exitCode=60 Oct 30 21:30:02.790555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.790169 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerDied","Data":"0208ee8ce1eff583dc50d8520995ca030f108bfb2a64601191094b9464a17644"} Oct 30 21:30:02.790555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.790397 4921 scope.go:122] "RemoveContainer" containerID="0208ee8ce1eff583dc50d8520995ca030f108bfb2a64601191094b9464a17644" Oct 30 21:30:02.792136 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.792109 4921 generic.go:356] "Generic (PLEG): container finished" podID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerID="cb02cc159d1fec8ab7c725e56f5824a3c652579413b5b866233020182a11eb48" exitCode=60 Oct 30 21:30:02.792252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.792204 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerDied","Data":"cb02cc159d1fec8ab7c725e56f5824a3c652579413b5b866233020182a11eb48"} Oct 30 21:30:02.792543 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.792502 4921 scope.go:122] "RemoveContainer" containerID="cb02cc159d1fec8ab7c725e56f5824a3c652579413b5b866233020182a11eb48" Oct 30 21:30:02.794192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.794175 4921 generic.go:356] "Generic (PLEG): container finished" podID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerID="11e256982860913a3bf00884f6f06fe321d919347f5216cc2338473c983d632d" exitCode=60 Oct 30 21:30:02.794300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.794257 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerDied","Data":"11e256982860913a3bf00884f6f06fe321d919347f5216cc2338473c983d632d"} Oct 30 21:30:02.794452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.794432 4921 scope.go:122] "RemoveContainer" containerID="11e256982860913a3bf00884f6f06fe321d919347f5216cc2338473c983d632d" Oct 30 21:30:02.800944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.800923 4921 generic.go:356] "Generic (PLEG): container finished" podID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerID="da21b592bcfa552985654e187dcd2e70f9edc9f10dece31e436543ee7e541627" exitCode=60 Oct 30 21:30:02.801041 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.800991 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerDied","Data":"da21b592bcfa552985654e187dcd2e70f9edc9f10dece31e436543ee7e541627"} Oct 30 21:30:02.801159 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:02.801147 4921 scope.go:122] "RemoveContainer" containerID="da21b592bcfa552985654e187dcd2e70f9edc9f10dece31e436543ee7e541627" Oct 30 21:30:03.812918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.812890 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerStarted","Data":"d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7"} Oct 30 21:30:03.813271 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.813161 4921 scope.go:122] "RemoveContainer" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" Oct 30 21:30:03.813512 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:03.813490 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch_odh-ai-gateway-infra(dbd90a2a-30e0-4951-9f0f-dbe14230846f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" Oct 30 21:30:03.814985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.814963 4921 generic.go:356] "Generic (PLEG): container finished" podID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerID="e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000" exitCode=60 Oct 30 21:30:03.815095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.815073 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerDied","Data":"e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000"} Oct 30 21:30:03.815156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.815131 4921 scope.go:122] "RemoveContainer" containerID="0208ee8ce1eff583dc50d8520995ca030f108bfb2a64601191094b9464a17644" Oct 30 21:30:03.815307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.815282 4921 scope.go:122] "RemoveContainer" containerID="e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000" Oct 30 21:30:03.815593 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:03.815569 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6_odh-ai-gateway-infra(724383d8-40ac-4dae-bcce-8e73ede1cc6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" Oct 30 21:30:03.817151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.817131 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerStarted","Data":"4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e"} Oct 30 21:30:03.817477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.817462 4921 scope.go:122] "RemoveContainer" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" Oct 30 21:30:03.817693 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:03.817671 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-chcv7_odh-ai-gateway-infra(1c82d16c-6e6f-43fe-9379-473b42fbe6b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" Oct 30 21:30:03.819064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.819044 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerStarted","Data":"ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f"} Oct 30 21:30:03.819289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:03.819275 4921 scope.go:122] "RemoveContainer" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" Oct 30 21:30:03.819488 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:03.819468 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2_odh-ai-gateway-infra(50311055-1eec-4dbb-bde6-b8e3ac556a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" Oct 30 21:30:04.828842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.828814 4921 scope.go:122] "RemoveContainer" containerID="e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000" Oct 30 21:30:04.829301 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:04.829103 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6_odh-ai-gateway-infra(724383d8-40ac-4dae-bcce-8e73ede1cc6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" Oct 30 21:30:04.830652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.830620 4921 generic.go:356] "Generic (PLEG): container finished" podID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" exitCode=60 Oct 30 21:30:04.830785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.830701 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerDied","Data":"4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e"} Oct 30 21:30:04.830937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.830913 4921 scope.go:122] "RemoveContainer" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" Oct 30 21:30:04.831020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.830991 4921 scope.go:122] "RemoveContainer" containerID="cb02cc159d1fec8ab7c725e56f5824a3c652579413b5b866233020182a11eb48" Oct 30 21:30:04.831170 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:04.831150 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-chcv7_odh-ai-gateway-infra(1c82d16c-6e6f-43fe-9379-473b42fbe6b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" Oct 30 21:30:04.833118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.833094 4921 generic.go:356] "Generic (PLEG): container finished" podID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" exitCode=60 Oct 30 21:30:04.833214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.833173 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerDied","Data":"ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f"} Oct 30 21:30:04.833431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.833411 4921 scope.go:122] "RemoveContainer" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" Oct 30 21:30:04.833664 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:04.833642 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2_odh-ai-gateway-infra(50311055-1eec-4dbb-bde6-b8e3ac556a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" Oct 30 21:30:04.838520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.838496 4921 generic.go:356] "Generic (PLEG): container finished" podID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" exitCode=60 Oct 30 21:30:04.838677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.838649 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerDied","Data":"d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7"} Oct 30 21:30:04.838808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.838790 4921 scope.go:122] "RemoveContainer" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" Oct 30 21:30:04.839073 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:04.839051 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch_odh-ai-gateway-infra(dbd90a2a-30e0-4951-9f0f-dbe14230846f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" Oct 30 21:30:04.844867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.844848 4921 scope.go:122] "RemoveContainer" containerID="11e256982860913a3bf00884f6f06fe321d919347f5216cc2338473c983d632d" Oct 30 21:30:04.857259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:04.857236 4921 scope.go:122] "RemoveContainer" containerID="da21b592bcfa552985654e187dcd2e70f9edc9f10dece31e436543ee7e541627" Oct 30 21:30:05.848668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:05.848642 4921 scope.go:122] "RemoveContainer" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" Oct 30 21:30:05.849094 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:05.848925 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-chcv7_odh-ai-gateway-infra(1c82d16c-6e6f-43fe-9379-473b42fbe6b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" Oct 30 21:30:05.850254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:05.850229 4921 scope.go:122] "RemoveContainer" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" Oct 30 21:30:05.850517 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:05.850495 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2_odh-ai-gateway-infra(50311055-1eec-4dbb-bde6-b8e3ac556a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" Oct 30 21:30:05.851862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:05.851846 4921 scope.go:122] "RemoveContainer" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" Oct 30 21:30:05.852025 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:05.852009 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch_odh-ai-gateway-infra(dbd90a2a-30e0-4951-9f0f-dbe14230846f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" Oct 30 21:30:07.539872 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:07.539796 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:30:08.340115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.340082 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:30:08.345376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.345359 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.435943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.435909 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436082 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.435966 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436082 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436012 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436124 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436153 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436180 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436337 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436412 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.436487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.436456 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.450689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.450661 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:30:08.520333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.520286 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:30:08.525748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.525726 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.537397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537376 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537408 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537442 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537592 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537628 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537773 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537750 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537766 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537812 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537846 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.537881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537873 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.538097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537889 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.538097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.537924 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.538097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.538079 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.538720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.538700 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.539806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.539783 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.539903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.539837 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.577366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.577340 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9ac5c5\"/\"distinct-w6-9ac5c5-kserve-self-signed-certs\"" Oct 30 21:30:08.579177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.579155 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:30:08.594527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.594472 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.594597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.594549 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.639363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639340 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jsr\" (UniqueName: \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.639458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639399 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.639458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639435 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.639537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639465 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.639570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639557 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.639633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.639618 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.658419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.658399 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.741933 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jsr\" (UniqueName: \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.742024 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.742079 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.742849 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.742956 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.743016 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.743410 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.743649 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.745354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.743705 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.746776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.746750 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.750361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.750340 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.798735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.798709 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jsr\" (UniqueName: \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\") pod \"distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:08.838017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:08.837992 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:09.161057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.160991 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:30:09.203229 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:09.203204 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bf56a4_259a_4327_9859_cf3e8bb85a26.slice/crio-9cd2f6f6615cda7dffb556d48fbc76909759fc38d6b935bb74759acd7b2e6261 WatchSource:0}: Error finding container 9cd2f6f6615cda7dffb556d48fbc76909759fc38d6b935bb74759acd7b2e6261: Status 404 returned error can't find the container with id 9cd2f6f6615cda7dffb556d48fbc76909759fc38d6b935bb74759acd7b2e6261 Oct 30 21:30:09.205528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.205495 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:09.205648 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.205570 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:09.205648 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.205615 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:09.221343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.221286 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:30:09.883337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.882942 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" event={"ID":"b6bf56a4-259a-4327-9859-cf3e8bb85a26","Type":"ContainerStarted","Data":"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef"} Oct 30 21:30:09.883337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.883000 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" event={"ID":"b6bf56a4-259a-4327-9859-cf3e8bb85a26","Type":"ContainerStarted","Data":"9cd2f6f6615cda7dffb556d48fbc76909759fc38d6b935bb74759acd7b2e6261"} Oct 30 21:30:09.885233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.885208 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" event={"ID":"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7","Type":"ContainerStarted","Data":"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24"} Oct 30 21:30:09.885426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:09.885239 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" event={"ID":"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7","Type":"ContainerStarted","Data":"1668444ef18ba1d1efebadd689d4ce5c9ba338f51ca4e77210a3ebdd5d5bba4c"} Oct 30 21:30:10.027343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.018480 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podStartSLOduration=2.018460873 podStartE2EDuration="2.018460873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:10.015796661 +0000 UTC m=+1123.998497125" watchObservedRunningTime="2026-10-30 21:30:10.018460873 +0000 UTC m=+1124.001161292" Oct 30 21:30:10.133329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.133247 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podStartSLOduration=2.133228211 podStartE2EDuration="2.133228211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:10.129160905 +0000 UTC m=+1124.111861323" watchObservedRunningTime="2026-10-30 21:30:10.133228211 +0000 UTC m=+1124.115928630" Oct 30 21:30:10.659039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.659010 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:10.660541 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.660508 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:30:10.660666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.660585 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 21:30:10.838865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:10.838833 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:11.659390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:11.659356 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:30:11.659779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:11.659406 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 21:30:12.658821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:12.658786 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:30:12.659006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:12.658851 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 21:30:13.659684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:13.659652 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:30:13.660172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:13.659703 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 21:30:14.659181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:14.659149 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 21:30:14.659399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:14.659195 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 21:30:15.161327 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.161280 4921 scope.go:122] "RemoveContainer" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" Oct 30 21:30:15.161728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.161375 4921 scope.go:122] "RemoveContainer" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" Oct 30 21:30:15.919857 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.919826 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:15.920633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.920601 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:15.921592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.921551 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:30:15.948964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.948931 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerStarted","Data":"0083678e8611288ee71ef1bf16f975372175c944c86f58650c2b900c6036d7de"} Oct 30 21:30:15.951572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:15.951543 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerStarted","Data":"86c799279adacbad5f68fbf0ab8cec8618e63858043fdf40d2520c5f194f3ff0"} Oct 30 21:30:16.960002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.959970 4921 generic.go:356] "Generic (PLEG): container finished" podID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerID="86c799279adacbad5f68fbf0ab8cec8618e63858043fdf40d2520c5f194f3ff0" exitCode=60 Oct 30 21:30:16.960495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.960041 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerDied","Data":"86c799279adacbad5f68fbf0ab8cec8618e63858043fdf40d2520c5f194f3ff0"} Oct 30 21:30:16.960495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.960098 4921 scope.go:122] "RemoveContainer" containerID="4caec1dc91c18beaece80fa0a4acb3f0550a5e5543df504b1fabc71136f7e46e" Oct 30 21:30:16.960495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.960437 4921 scope.go:122] "RemoveContainer" containerID="86c799279adacbad5f68fbf0ab8cec8618e63858043fdf40d2520c5f194f3ff0" Oct 30 21:30:16.960736 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:16.960715 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-chcv7_odh-ai-gateway-infra(1c82d16c-6e6f-43fe-9379-473b42fbe6b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" Oct 30 21:30:16.962362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.962342 4921 generic.go:356] "Generic (PLEG): container finished" podID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerID="0083678e8611288ee71ef1bf16f975372175c944c86f58650c2b900c6036d7de" exitCode=60 Oct 30 21:30:16.962481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.962419 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerDied","Data":"0083678e8611288ee71ef1bf16f975372175c944c86f58650c2b900c6036d7de"} Oct 30 21:30:16.962722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.962706 4921 scope.go:122] "RemoveContainer" containerID="0083678e8611288ee71ef1bf16f975372175c944c86f58650c2b900c6036d7de" Oct 30 21:30:16.962947 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:16.962928 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch_odh-ai-gateway-infra(dbd90a2a-30e0-4951-9f0f-dbe14230846f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" Oct 30 21:30:16.973699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:16.973676 4921 scope.go:122] "RemoveContainer" containerID="d5b9a8b88a75e5e2beba5e0b8b6ae4a2a21fe8663400f763bda6ff5a6603edb7" Oct 30 21:30:17.160990 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:17.160963 4921 scope.go:122] "RemoveContainer" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" Oct 30 21:30:17.210690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:17.210635 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:17.326964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:17.326936 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:17.973469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:17.973440 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerStarted","Data":"358b12f9bc839f9bcd190c51531259f9422376695ea9cfed74f7ad13c112458c"} Oct 30 21:30:18.161428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.161409 4921 scope.go:122] "RemoveContainer" containerID="e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000" Oct 30 21:30:18.219831 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.219804 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:18.221841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.221820 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:18.273537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273485 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\" (UniqueName: \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\") pod \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " Oct 30 21:30:18.273641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273628 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\") pod \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " Oct 30 21:30:18.273690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273664 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\") pod \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\" (UID: \"1c82d16c-6e6f-43fe-9379-473b42fbe6b4\") " Oct 30 21:30:18.273743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273694 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\" (UniqueName: \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\") pod \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\" (UID: \"dbd90a2a-30e0-4951-9f0f-dbe14230846f\") " Oct 30 21:30:18.274054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273936 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca" pod "1c82d16c-6e6f-43fe-9379-473b42fbe6b4" (UID: "1c82d16c-6e6f-43fe-9379-473b42fbe6b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:18.274054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.273999 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca" pod "dbd90a2a-30e0-4951-9f0f-dbe14230846f" (UID: "dbd90a2a-30e0-4951-9f0f-dbe14230846f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:18.275573 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.275553 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69" pod "1c82d16c-6e6f-43fe-9379-473b42fbe6b4" (UID: "1c82d16c-6e6f-43fe-9379-473b42fbe6b4"). InnerVolumeSpecName "kube-api-access-2sj69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:18.275746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.275727 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5" pod "dbd90a2a-30e0-4951-9f0f-dbe14230846f" (UID: "dbd90a2a-30e0-4951-9f0f-dbe14230846f"). InnerVolumeSpecName "kube-api-access-hbhz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:18.375051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.375018 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd90a2a-30e0-4951-9f0f-dbe14230846f-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:18.375051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.375046 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:18.375198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.375062 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbhz5\" (UniqueName: \"kubernetes.io/projected/dbd90a2a-30e0-4951-9f0f-dbe14230846f-kube-api-access-hbhz5\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:18.375198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.375076 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sj69\" (UniqueName: \"kubernetes.io/projected/1c82d16c-6e6f-43fe-9379-473b42fbe6b4-kube-api-access-2sj69\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:18.839193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.839164 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:18.851672 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.851648 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:18.985516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.985481 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerStarted","Data":"e05e826b3a4b12bba1da784aeec80f148372b430964e5439ee7e6e422a925517"} Oct 30 21:30:18.987230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.987195 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" event={"ID":"1c82d16c-6e6f-43fe-9379-473b42fbe6b4","Type":"ContainerDied","Data":"fa37b5723c9fb64e0e7418d75f0e650febb3b8620065e6750bae54800fe3fb19"} Oct 30 21:30:18.987479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.987461 4921 scope.go:122] "RemoveContainer" containerID="86c799279adacbad5f68fbf0ab8cec8618e63858043fdf40d2520c5f194f3ff0" Oct 30 21:30:18.987671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.987279 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7" Oct 30 21:30:18.991738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.991709 4921 generic.go:356] "Generic (PLEG): container finished" podID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerID="358b12f9bc839f9bcd190c51531259f9422376695ea9cfed74f7ad13c112458c" exitCode=60 Oct 30 21:30:18.991941 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.991923 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerDied","Data":"358b12f9bc839f9bcd190c51531259f9422376695ea9cfed74f7ad13c112458c"} Oct 30 21:30:18.992354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:18.992334 4921 scope.go:122] "RemoveContainer" containerID="358b12f9bc839f9bcd190c51531259f9422376695ea9cfed74f7ad13c112458c" Oct 30 21:30:18.992732 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:18.992708 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2_odh-ai-gateway-infra(50311055-1eec-4dbb-bde6-b8e3ac556a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" Oct 30 21:30:19.004952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.004933 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" Oct 30 21:30:19.006474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.005034 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch" event={"ID":"dbd90a2a-30e0-4951-9f0f-dbe14230846f","Type":"ContainerDied","Data":"77368c095d3f3a0db2dce290ece1a2c4b85ca4153a5e1836ee10ac18a43d9433"} Oct 30 21:30:19.015021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.015005 4921 scope.go:122] "RemoveContainer" containerID="ac15706e49acfe07f09cd94995d6cd6077ee6a9f15bfe57532ecd6267b79100f" Oct 30 21:30:19.021720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.021702 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:30:19.046756 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.046730 4921 scope.go:122] "RemoveContainer" containerID="0083678e8611288ee71ef1bf16f975372175c944c86f58650c2b900c6036d7de" Oct 30 21:30:19.353631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.353598 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:19.440388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.440176 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:19.447062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.447033 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-chcv7"] Oct 30 21:30:19.551770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.551739 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:19.555055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.555034 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f52c3-29889930-7qvch"] Oct 30 21:30:19.727448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.727418 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:30:19.728231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728213 4921 cpu_manager.go:414] "containerMap: removing container" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728234 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728252 4921 cpu_manager.go:414] "containerMap: removing container" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728260 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728282 4921 cpu_manager.go:414] "containerMap: removing container" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728291 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728356 4921 cpu_manager.go:414] "containerMap: removing container" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728364 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728505 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728517 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:19.728534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728531 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728539 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.728622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.728548 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:19.733812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.733794 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.785558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.785498 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-be9451\"/\"openshift-service-ca.crt\"" Oct 30 21:30:19.785717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.785597 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-be9451\"/\"kube-root-ca.crt\"" Oct 30 21:30:19.785717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.785643 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-be9451\"/\"default-dockercfg-xpqsf\"" Oct 30 21:30:19.785843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.785747 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-be9451\"/\"distinct-w0-be9451-kserve-self-signed-certs\"" Oct 30 21:30:19.787262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.787239 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:30:19.791283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791261 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.791415 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791287 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.791478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791431 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.791478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791457 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnpc\" (UniqueName: \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.791590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791545 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.791590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.791573 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893186 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893251 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893413 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893439 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893572 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893604 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnpc\" (UniqueName: \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.893861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.893753 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.895424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.894008 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.895424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.894592 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.896952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.896926 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.897763 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.897739 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:19.949995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:19.949967 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnpc\" (UniqueName: \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\") pod \"distinct-w0-be9451-kserve-6ccccdc5cf-fnchh\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:20.016249 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.016170 4921 generic.go:356] "Generic (PLEG): container finished" podID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerID="e05e826b3a4b12bba1da784aeec80f148372b430964e5439ee7e6e422a925517" exitCode=60 Oct 30 21:30:20.016677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.016251 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerDied","Data":"e05e826b3a4b12bba1da784aeec80f148372b430964e5439ee7e6e422a925517"} Oct 30 21:30:20.016677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.016354 4921 scope.go:122] "RemoveContainer" containerID="e322a683906bb927663ef61c0fb6e3e472527f99b7522f7754d2e09891400000" Oct 30 21:30:20.016789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.016715 4921 scope.go:122] "RemoveContainer" containerID="e05e826b3a4b12bba1da784aeec80f148372b430964e5439ee7e6e422a925517" Oct 30 21:30:20.017046 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:20.017021 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6_odh-ai-gateway-infra(724383d8-40ac-4dae-bcce-8e73ede1cc6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" Oct 30 21:30:20.048843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.048575 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:20.159281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.155915 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:20.213713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.213541 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" path="/var/lib/kubelet/pods/1c82d16c-6e6f-43fe-9379-473b42fbe6b4/volumes" Oct 30 21:30:20.227783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.227760 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" path="/var/lib/kubelet/pods/dbd90a2a-30e0-4951-9f0f-dbe14230846f/volumes" Oct 30 21:30:20.297588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.297559 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:20.308962 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.308938 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\") pod \"50311055-1eec-4dbb-bde6-b8e3ac556a82\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " Oct 30 21:30:20.309093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.308972 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\" (UniqueName: \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\") pod \"50311055-1eec-4dbb-bde6-b8e3ac556a82\" (UID: \"50311055-1eec-4dbb-bde6-b8e3ac556a82\") " Oct 30 21:30:20.309307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.309272 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca" pod "50311055-1eec-4dbb-bde6-b8e3ac556a82" (UID: "50311055-1eec-4dbb-bde6-b8e3ac556a82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:20.309640 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.309621 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50311055-1eec-4dbb-bde6-b8e3ac556a82-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.314348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.312754 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n" pod "50311055-1eec-4dbb-bde6-b8e3ac556a82" (UID: "50311055-1eec-4dbb-bde6-b8e3ac556a82"). InnerVolumeSpecName "kube-api-access-68x5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:20.411335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.411153 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68x5n\" (UniqueName: \"kubernetes.io/projected/50311055-1eec-4dbb-bde6-b8e3ac556a82-kube-api-access-68x5n\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.558825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:20.558795 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:30:20.561475 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:20.561441 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb422e0_1506_4ccb_84d1_e29858c3b9cd.slice/crio-51eeeb404d52b97fe3f75faf52f5f1c68f951289fdeff31e622e330b02cc849e WatchSource:0}: Error finding container 51eeeb404d52b97fe3f75faf52f5f1c68f951289fdeff31e622e330b02cc849e: Status 404 returned error can't find the container with id 51eeeb404d52b97fe3f75faf52f5f1c68f951289fdeff31e622e330b02cc849e Oct 30 21:30:21.052838 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.052802 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" event={"ID":"50311055-1eec-4dbb-bde6-b8e3ac556a82","Type":"ContainerDied","Data":"dc232fc2797ce6f40de84598a60912a487e1245b61fb19cc07a037279d6d9b8c"} Oct 30 21:30:21.053212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.052875 4921 scope.go:122] "RemoveContainer" containerID="358b12f9bc839f9bcd190c51531259f9422376695ea9cfed74f7ad13c112458c" Oct 30 21:30:21.053212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.052974 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2" Oct 30 21:30:21.129522 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.125757 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" event={"ID":"3eb422e0-1506-4ccb-84d1-e29858c3b9cd","Type":"ContainerStarted","Data":"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf"} Oct 30 21:30:21.129522 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.125810 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" event={"ID":"3eb422e0-1506-4ccb-84d1-e29858c3b9cd","Type":"ContainerStarted","Data":"51eeeb404d52b97fe3f75faf52f5f1c68f951289fdeff31e622e330b02cc849e"} Oct 30 21:30:21.278183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.278160 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:21.303925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.302859 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:21.308509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.308482 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889930-nz7d2"] Oct 30 21:30:21.335334 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.335292 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\" (UniqueName: \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\") pod \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " Oct 30 21:30:21.335493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.335472 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\") pod \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\" (UID: \"724383d8-40ac-4dae-bcce-8e73ede1cc6d\") " Oct 30 21:30:21.336003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.335978 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca" pod "724383d8-40ac-4dae-bcce-8e73ede1cc6d" (UID: "724383d8-40ac-4dae-bcce-8e73ede1cc6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:21.338068 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.338042 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws" pod "724383d8-40ac-4dae-bcce-8e73ede1cc6d" (UID: "724383d8-40ac-4dae-bcce-8e73ede1cc6d"). InnerVolumeSpecName "kube-api-access-zvhws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:21.416530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.416476 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podStartSLOduration=2.416462831 podStartE2EDuration="2.416462831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:21.412677258 +0000 UTC m=+1135.395377688" watchObservedRunningTime="2026-10-30 21:30:21.416462831 +0000 UTC m=+1135.399163247" Oct 30 21:30:21.436607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.436577 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhws\" (UniqueName: \"kubernetes.io/projected/724383d8-40ac-4dae-bcce-8e73ede1cc6d-kube-api-access-zvhws\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:21.436607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:21.436602 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/724383d8-40ac-4dae-bcce-8e73ede1cc6d-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:22.049947 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.049917 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:22.132878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.132851 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" Oct 30 21:30:22.132878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.132858 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6" event={"ID":"724383d8-40ac-4dae-bcce-8e73ede1cc6d","Type":"ContainerDied","Data":"daeb87cc792f77e74893b9cbce53a8ac7968f01f8bf9136d103b0e418bf9a4cb"} Oct 30 21:30:22.133382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.132920 4921 scope.go:122] "RemoveContainer" containerID="e05e826b3a4b12bba1da784aeec80f148372b430964e5439ee7e6e422a925517" Oct 30 21:30:22.168486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.168465 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" path="/var/lib/kubelet/pods/50311055-1eec-4dbb-bde6-b8e3ac556a82/volumes" Oct 30 21:30:22.296430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.296405 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:22.345515 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:22.345442 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0a4eab-29889930-tjlz6"] Oct 30 21:30:24.167557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:24.167529 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" path="/var/lib/kubelet/pods/724383d8-40ac-4dae-bcce-8e73ede1cc6d/volumes" Oct 30 21:30:30.049866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:30.049829 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:30.062237 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:30.062214 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:30.227489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:30.227460 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:30:37.539823 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:30:37.539788 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:30:53.046786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.046744 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:30:53.047571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047551 4921 cpu_manager.go:414] "containerMap: removing container" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047573 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047603 4921 cpu_manager.go:414] "containerMap: removing container" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047611 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047630 4921 cpu_manager.go:414] "containerMap: removing container" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047637 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047651 4921 cpu_manager.go:414] "containerMap: removing container" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047658 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:53.047679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047680 4921 cpu_manager.go:414] "containerMap: removing container" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047688 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047703 4921 cpu_manager.go:414] "containerMap: removing container" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047711 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd90a2a-30e0-4951-9f0f-dbe14230846f" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047725 4921 cpu_manager.go:414] "containerMap: removing container" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047731 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047745 4921 cpu_manager.go:414] "containerMap: removing container" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047752 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047876 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c82d16c-6e6f-43fe-9379-473b42fbe6b4" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047890 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047902 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047912 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="50311055-1eec-4dbb-bde6-b8e3ac556a82" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047926 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.048051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.047938 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:30:53.051656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.051634 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.159639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.159597 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:30:53.187748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.187712 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklqd\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.187897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.187761 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.187897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.187849 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.187897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.187865 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.187995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.187906 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.188033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.188012 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.188065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.188049 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.188153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.188138 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.188307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.188293 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289119 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklqd\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289172 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289212 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289234 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289300 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289385 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289412 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289460 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289613 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289666 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289718 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.289998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.289973 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.290052 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.290008 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.290430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.290406 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.291680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.291651 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.292040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.292016 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.345505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.345442 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklqd\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.345505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.345444 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.364932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.364910 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:53.797570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.797539 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:30:53.798246 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:30:53.798221 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb430dd5c_6f2d_4034_8d7b_08f78e3ddeb5.slice/crio-6808b870164994dfaab031926b4c554ff78fd66b3382e6162ef7a7111359772d WatchSource:0}: Error finding container 6808b870164994dfaab031926b4c554ff78fd66b3382e6162ef7a7111359772d: Status 404 returned error can't find the container with id 6808b870164994dfaab031926b4c554ff78fd66b3382e6162ef7a7111359772d Oct 30 21:30:53.800697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.800665 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:53.800774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.800729 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:53.800774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:53.800764 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:30:54.419288 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:54.419246 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" event={"ID":"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5","Type":"ContainerStarted","Data":"2ec0902066a4095e31c473dccba37130cb05dadee3ba375a7a23ac1a7f41d923"} Oct 30 21:30:54.419679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:54.419310 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" event={"ID":"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5","Type":"ContainerStarted","Data":"6808b870164994dfaab031926b4c554ff78fd66b3382e6162ef7a7111359772d"} Oct 30 21:30:54.541285 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:54.541224 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podStartSLOduration=2.541205158 podStartE2EDuration="2.541205158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:54.539496987 +0000 UTC m=+1168.522197406" watchObservedRunningTime="2026-10-30 21:30:54.541205158 +0000 UTC m=+1168.523905576" Oct 30 21:30:55.366012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:55.365984 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:30:55.367878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:55.367853 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:30:55.368001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:55.367896 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:30:56.365263 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:56.365231 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:30:56.365674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:56.365304 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:30:57.365682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:57.365645 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:30:57.366081 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:57.365701 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:30:58.365296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:58.365260 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:30:58.365476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:58.365339 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:30:59.365683 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:59.365637 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:30:59.366178 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:30:59.365703 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:31:00.365755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:00.365721 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:31:00.366125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:00.365770 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 21:31:01.064410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.064374 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:31:01.066031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.066005 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="724383d8-40ac-4dae-bcce-8e73ede1cc6d" containerName="cleanup" Oct 30 21:31:01.074502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.074478 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098359 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098401 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098443 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8s92\" (UniqueName: \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098474 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098506 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.098736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.098537 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.125263 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.125238 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:31:01.170536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.170510 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8ce31a\"/\"distinct-2-w1-8ce31a-kserve-self-signed-certs\"" Oct 30 21:31:01.200107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200081 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8s92\" (UniqueName: \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.200235 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200159 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.200235 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200211 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.200379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200244 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.200533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200506 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.200623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.200541 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.201025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.201005 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.201827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.201805 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.202021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.202003 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.206823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.206800 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.207704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.207680 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.258226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.258197 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8s92\" (UniqueName: \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\") pod \"distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.370046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.369974 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:31:01.370520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.370410 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:31:01.371000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.370981 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:31:01.391361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.391338 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:01.796401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:01.796369 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:31:01.796588 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:31:01.796567 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f14a12_67ba_46fe_a785_f30290653466.slice/crio-d72d4a04227129df14d54c414c84ac378bb984c607c79a2e54bf1bf557204d17 WatchSource:0}: Error finding container d72d4a04227129df14d54c414c84ac378bb984c607c79a2e54bf1bf557204d17: Status 404 returned error can't find the container with id d72d4a04227129df14d54c414c84ac378bb984c607c79a2e54bf1bf557204d17 Oct 30 21:31:02.490935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:02.490902 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" event={"ID":"11f14a12-67ba-46fe-a785-f30290653466","Type":"ContainerStarted","Data":"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a"} Oct 30 21:31:02.491385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:02.490948 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" event={"ID":"11f14a12-67ba-46fe-a785-f30290653466","Type":"ContainerStarted","Data":"d72d4a04227129df14d54c414c84ac378bb984c607c79a2e54bf1bf557204d17"} Oct 30 21:31:02.608639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:02.608578 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podStartSLOduration=1.608560397 podStartE2EDuration="1.608560397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:02.604526978 +0000 UTC m=+1176.587227463" watchObservedRunningTime="2026-10-30 21:31:02.608560397 +0000 UTC m=+1176.591260817" Oct 30 21:31:03.391966 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:03.391936 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:05.425590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.425553 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ldlfp-ztts9"] Oct 30 21:31:05.429720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.429702 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.485477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.485449 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ldlfp-ztts9"] Oct 30 21:31:05.530708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.530684 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9zfm5\"" Oct 30 21:31:05.564135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.564100 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzw82\" (UniqueName: \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.564253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.564208 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.564253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.564236 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.564431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.564309 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.665867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.665837 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.665987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.665875 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.666040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.666001 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.666185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.666170 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzw82\" (UniqueName: \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.666414 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.666393 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.666582 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.666568 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.666651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.666632 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.723529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.723498 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzw82\" (UniqueName: \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\") pod \"periodic-gathering-ldlfp-ztts9\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:05.744191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:05.744166 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:31:06.117576 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:31:06.117548 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7078a48_6135_462f_848c_808eb51810d6.slice/crio-2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247 WatchSource:0}: Error finding container 2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247: Status 404 returned error can't find the container with id 2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247 Oct 30 21:31:06.117947 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:06.117925 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ldlfp-ztts9"] Oct 30 21:31:06.524302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:06.524266 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" event={"ID":"d7078a48-6135-462f-848c-808eb51810d6","Type":"ContainerStarted","Data":"1816a700843a7747f4ed95fa4f845a49fbba4a5dc9b134b9ca6e576d142d1ffe"} Oct 30 21:31:06.524709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:06.524336 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" event={"ID":"d7078a48-6135-462f-848c-808eb51810d6","Type":"ContainerStarted","Data":"2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247"} Oct 30 21:31:06.641829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:06.641783 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" podStartSLOduration=1.64177016 podStartE2EDuration="1.64177016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:06.637073644 +0000 UTC m=+1180.619774062" watchObservedRunningTime="2026-10-30 21:31:06.64177016 +0000 UTC m=+1180.624470599" Oct 30 21:31:07.539836 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:31:07.539806 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:31:11.391810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:11.391763 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:11.403973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:11.403948 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:11.572050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:11.572020 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:31:26.328479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:26.328446 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:31:26.340153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:26.340130 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:31:26.341228 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:26.341209 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:31:26.353097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:26.353081 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:31:37.545527 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:31:37.545408 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:31:54.185968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.185933 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:31:54.189976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.189957 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.282608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282562 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.282806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282631 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.282806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282662 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.282806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282687 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.282806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282719 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.283038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282899 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.283038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.282948 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.283038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.283016 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfwq\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.283197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.283070 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.283423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.283396 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:31:54.383934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.383899 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.383941 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.383985 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfwq\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384013 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384060 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384108 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384134 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384157 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384187 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384310 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384469 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384683 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384592 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.384683 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.384640 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.385032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.385015 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.386405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.386387 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.386639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.386622 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.440990 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.440917 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.441664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.441643 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfwq\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.504111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.504078 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:54.939716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.939680 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:31:54.940255 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:31:54.940231 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048998df_7505_4e39_b56e_6a1f4d131fb3.slice/crio-f84b451640128f481bfb45ce2b20366342c633d78a08bb9769ce780305e36d1d WatchSource:0}: Error finding container f84b451640128f481bfb45ce2b20366342c633d78a08bb9769ce780305e36d1d: Status 404 returned error can't find the container with id f84b451640128f481bfb45ce2b20366342c633d78a08bb9769ce780305e36d1d Oct 30 21:31:54.942800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.942765 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:31:54.942879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.942832 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:31:54.942879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:54.942862 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:31:55.933012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:55.932969 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" event={"ID":"048998df-7505-4e39-b56e-6a1f4d131fb3","Type":"ContainerStarted","Data":"20e4e4320fbde60c71abd348c1913366c5c8d7f9b4fa31afb0ea0384c12e38d6"} Oct 30 21:31:55.933561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:55.933026 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" event={"ID":"048998df-7505-4e39-b56e-6a1f4d131fb3","Type":"ContainerStarted","Data":"f84b451640128f481bfb45ce2b20366342c633d78a08bb9769ce780305e36d1d"} Oct 30 21:31:56.132338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:56.131738 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podStartSLOduration=2.131716168 podStartE2EDuration="2.131716168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:56.121684901 +0000 UTC m=+1230.104385321" watchObservedRunningTime="2026-10-30 21:31:56.131716168 +0000 UTC m=+1230.114416588" Oct 30 21:31:56.506339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:56.506120 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:31:56.506339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:56.506291 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:31:56.506652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:56.506370 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:31:57.507342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:57.505792 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:31:57.507342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:57.505859 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:31:58.504680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:58.504640 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:31:58.504967 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:58.504936 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:31:59.504790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:59.504745 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:31:59.505446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:31:59.505413 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:32:00.514481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:00.514441 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:32:00.515129 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:00.515092 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:32:01.504779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:01.504734 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:32:01.505009 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:01.504849 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:32:02.509955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:02.509921 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:32:02.510458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:02.510346 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:32:02.511058 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:02.511039 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:32:07.541846 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:32:07.541799 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:32:07.957368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:07.957290 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:32:07.962307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:07.962282 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.070042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.069997 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:32:08.114997 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.114954 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115037 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115125 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115154 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115302 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115393 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw96k\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115453 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115575 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.115649 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.115633 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.216766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216673 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.216766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216730 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.216766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216758 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216808 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216844 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw96k\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216901 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.216994 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217076 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217116 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217136 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217435 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217449 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217726 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217698 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.217922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.217901 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.219920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.219876 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.220034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.219896 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.273201 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.273166 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.273385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.273221 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw96k\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.279046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.279024 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:08.717893 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.717850 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:32:08.718764 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:32:08.718723 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2264b3da_491c_44b8_b21e_9a1fecf33f10.slice/crio-7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645 WatchSource:0}: Error finding container 7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645: Status 404 returned error can't find the container with id 7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645 Oct 30 21:32:08.720915 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.720883 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:08.720995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.720950 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:08.720995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:08.720977 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:32:09.047759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.047665 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" event={"ID":"2264b3da-491c-44b8-b21e-9a1fecf33f10","Type":"ContainerStarted","Data":"8eb80001e2b12bfdd43b840c07db76b806c54242752d4c649bccc8a7e948f5f1"} Oct 30 21:32:09.047759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.047721 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" event={"ID":"2264b3da-491c-44b8-b21e-9a1fecf33f10","Type":"ContainerStarted","Data":"7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645"} Oct 30 21:32:09.171810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.171755 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podStartSLOduration=2.171739905 podStartE2EDuration="2.171739905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:09.165253711 +0000 UTC m=+1243.147954128" watchObservedRunningTime="2026-10-30 21:32:09.171739905 +0000 UTC m=+1243.154440322" Oct 30 21:32:09.279858 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.279824 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:09.281570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.281541 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:09.281755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:09.281612 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:10.279869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:10.279824 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:10.280256 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:10.279894 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:11.280106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:11.280063 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:11.280668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:11.280120 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:12.280133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:12.280090 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:12.280666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:12.280152 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:13.279975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:13.279936 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:13.280150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:13.279991 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:14.279871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:14.279827 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:14.280262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:14.279892 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:15.280020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:15.279976 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:32:15.280463 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:15.280059 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:32:16.283953 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:16.283924 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:16.284444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:16.284425 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:16.285047 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:16.285026 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:32:37.541845 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:32:37.541802 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:32:43.366012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:43.365974 4921 generic.go:356] "Generic (PLEG): container finished" podID="d7078a48-6135-462f-848c-808eb51810d6" containerID="1816a700843a7747f4ed95fa4f845a49fbba4a5dc9b134b9ca6e576d142d1ffe" exitCode=0 Oct 30 21:32:43.366556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:43.366045 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" event={"ID":"d7078a48-6135-462f-848c-808eb51810d6","Type":"ContainerDied","Data":"1816a700843a7747f4ed95fa4f845a49fbba4a5dc9b134b9ca6e576d142d1ffe"} Oct 30 21:32:44.540531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.540501 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:32:44.592257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592220 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\") pod \"d7078a48-6135-462f-848c-808eb51810d6\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " Oct 30 21:32:44.592479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592280 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\") pod \"d7078a48-6135-462f-848c-808eb51810d6\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " Oct 30 21:32:44.592479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592355 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\") pod \"d7078a48-6135-462f-848c-808eb51810d6\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " Oct 30 21:32:44.592602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592480 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\" (UniqueName: \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\") pod \"d7078a48-6135-462f-848c-808eb51810d6\" (UID: \"d7078a48-6135-462f-848c-808eb51810d6\") " Oct 30 21:32:44.592755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592732 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path" pod "d7078a48-6135-462f-848c-808eb51810d6" (UID: "d7078a48-6135-462f-848c-808eb51810d6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:44.592794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592749 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle" pod "d7078a48-6135-462f-848c-808eb51810d6" (UID: "d7078a48-6135-462f-848c-808eb51810d6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:44.592794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.592768 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle" pod "d7078a48-6135-462f-848c-808eb51810d6" (UID: "d7078a48-6135-462f-848c-808eb51810d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:44.593018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.593002 4921 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-service-ca-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.593056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.593022 4921 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d7078a48-6135-462f-848c-808eb51810d6-archives-path\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.593056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.593031 4921 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7078a48-6135-462f-848c-808eb51810d6-trusted-ca-bundle\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:44.594830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.594811 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82" pod "d7078a48-6135-462f-848c-808eb51810d6" (UID: "d7078a48-6135-462f-848c-808eb51810d6"). InnerVolumeSpecName "kube-api-access-gzw82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:44.694735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:44.694639 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzw82\" (UniqueName: \"kubernetes.io/projected/d7078a48-6135-462f-848c-808eb51810d6-kube-api-access-gzw82\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.381450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:45.381422 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" Oct 30 21:32:45.381656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:45.381415 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ldlfp-ztts9" event={"ID":"d7078a48-6135-462f-848c-808eb51810d6","Type":"ContainerDied","Data":"2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247"} Oct 30 21:32:45.381656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:45.381554 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2246cc2409d18bab53d7b17685b968949bfb94dd5ab27a6af7739e0c24f15247" Oct 30 21:32:48.699398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.699362 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:32:48.699955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.699941 4921 cpu_manager.go:414] "containerMap: removing container" podUID="d7078a48-6135-462f-848c-808eb51810d6" containerName="insights-gathering" Oct 30 21:32:48.699999 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.699955 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7078a48-6135-462f-848c-808eb51810d6" containerName="insights-gathering" Oct 30 21:32:48.700087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.700078 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7078a48-6135-462f-848c-808eb51810d6" containerName="insights-gathering" Oct 30 21:32:48.703706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.703689 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.745586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.745549 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.745754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.745671 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.745754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.745732 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnlgk\" (UniqueName: \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.745929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.745906 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.746202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.746175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.746335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.746219 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.805703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.805670 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-38b832\"/\"kube-root-ca.crt\"" Oct 30 21:32:48.805896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.805746 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-38b832\"/\"default-dockercfg-289hb\"" Oct 30 21:32:48.805896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.805768 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-38b832\"/\"openshift-service-ca.crt\"" Oct 30 21:32:48.805896 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.805805 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-38b832\"/\"distinct-2-w2-38b832-kserve-self-signed-certs\"" Oct 30 21:32:48.810038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.810020 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:32:48.847229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847194 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnlgk\" (UniqueName: \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847391 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847495 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847530 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847585 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847626 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.847970 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847933 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.848124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.847939 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.848124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.848003 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.849733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.849705 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.850193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.850170 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:48.904510 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:48.904469 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnlgk\" (UniqueName: \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\") pod \"distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:49.016226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:49.016142 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:49.421359 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:32:49.421329 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c98345b_c8d0_451f_8f01_47fa2499eb19.slice/crio-b54493122eb863a1d045946a230fde33bf5a3a43b4c0e6221e19d724ed0c688a WatchSource:0}: Error finding container b54493122eb863a1d045946a230fde33bf5a3a43b4c0e6221e19d724ed0c688a: Status 404 returned error can't find the container with id b54493122eb863a1d045946a230fde33bf5a3a43b4c0e6221e19d724ed0c688a Oct 30 21:32:49.423641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:49.423611 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:32:50.418537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:50.418497 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" event={"ID":"2c98345b-c8d0-451f-8f01-47fa2499eb19","Type":"ContainerStarted","Data":"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86"} Oct 30 21:32:50.419006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:50.418545 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" event={"ID":"2c98345b-c8d0-451f-8f01-47fa2499eb19","Type":"ContainerStarted","Data":"b54493122eb863a1d045946a230fde33bf5a3a43b4c0e6221e19d724ed0c688a"} Oct 30 21:32:50.535917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:50.535871 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podStartSLOduration=2.535854301 podStartE2EDuration="2.535854301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:50.532144826 +0000 UTC m=+1284.514845245" watchObservedRunningTime="2026-10-30 21:32:50.535854301 +0000 UTC m=+1284.518554720" Oct 30 21:32:51.017691 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:51.017659 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:59.016964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:59.016928 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:59.030101 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:59.030076 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:32:59.495971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:32:59.495939 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:33:07.540914 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:33:07.540835 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:33:22.051108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:22.051074 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9"] Oct 30 21:33:22.099796 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:22.099767 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6c35d7-cgqk9"] Oct 30 21:33:22.169272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:22.169244 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6cb422-0d2e-4500-ad58-11f0548b1513" path="/var/lib/kubelet/pods/cf6cb422-0d2e-4500-ad58-11f0548b1513/volumes" Oct 30 21:33:33.608509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:33.608460 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:33:33.952361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:33.951519 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:33:37.540973 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:33:37.540928 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:33:44.256144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.256110 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:33:44.260282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.260263 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.302990 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.303055 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.303185 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5tb\" (UniqueName: \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.303241 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.303370 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.303548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.303531 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.320071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.320036 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:33:44.363144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.363114 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f52c3\"/\"distinct-2-w3-9f52c3-kserve-self-signed-certs\"" Oct 30 21:33:44.405190 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.405140 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5tb\" (UniqueName: \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.407287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.407252 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.407538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.407491 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.410296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.410269 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.410730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.410664 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.410849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.410756 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.410849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.410844 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.411200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.411182 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.411609 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.411586 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.413687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.413662 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.432693 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.432659 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.462367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.462332 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5tb\" (UniqueName: \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\") pod \"distinct-2-w3-9f52c3-kserve-86464777bc-kdns7\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.572730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.572647 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:44.950074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:44.950048 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:33:44.951615 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:33:44.951592 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8d14b8_f2d6_4076_907f_dd31236d9fc5.slice/crio-98b7f000fdc750ea167972f39785e071eb9a6978d4940bbb2eced6e825a65d26 WatchSource:0}: Error finding container 98b7f000fdc750ea167972f39785e071eb9a6978d4940bbb2eced6e825a65d26: Status 404 returned error can't find the container with id 98b7f000fdc750ea167972f39785e071eb9a6978d4940bbb2eced6e825a65d26 Oct 30 21:33:45.947309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:45.947274 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" event={"ID":"fd8d14b8-f2d6-4076-907f-dd31236d9fc5","Type":"ContainerStarted","Data":"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c"} Oct 30 21:33:45.947309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:45.947331 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" event={"ID":"fd8d14b8-f2d6-4076-907f-dd31236d9fc5","Type":"ContainerStarted","Data":"98b7f000fdc750ea167972f39785e071eb9a6978d4940bbb2eced6e825a65d26"} Oct 30 21:33:46.052301 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:46.052268 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc"] Oct 30 21:33:46.055758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:46.055735 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5881bb19-x99nc"] Oct 30 21:33:46.100915 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:46.100867 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podStartSLOduration=2.100854137 podStartE2EDuration="2.100854137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:46.097623876 +0000 UTC m=+1340.080324287" watchObservedRunningTime="2026-10-30 21:33:46.100854137 +0000 UTC m=+1340.083554555" Oct 30 21:33:46.171717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:46.171677 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9254724c-69c0-4fad-a626-4365e9b75103" path="/var/lib/kubelet/pods/9254724c-69c0-4fad-a626-4365e9b75103/volumes" Oct 30 21:33:46.573142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:46.573109 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:54.572900 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:54.572864 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:54.584788 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:54.584763 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:33:55.030217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:33:55.030189 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:34:06.084347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:06.083704 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:07.540578 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:34:07.540526 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:34:09.208013 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.207981 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:34:09.213972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.213940 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.272805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.272769 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:34:09.292416 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292385 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292434 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292563 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292596 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292736 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292854 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpmj\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.292971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292958 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.293017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292977 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.293017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.292993 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394164 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpmj\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394248 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394264 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394280 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394303 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394365 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.394409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394402 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394424 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394516 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394760 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394896 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394932 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.394959 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.395226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.395133 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.396560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.396541 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.396844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.396827 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.453949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.453911 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.454141 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.454115 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpmj\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.530087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.529983 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:09.968528 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:34:09.968491 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586eea25_2b06_4402_9c43_ebfe2bbe5caa.slice/crio-78f2690cbfcb4b7b1e1ebc02a2252fe7a01232b169ce1e71c8ed3546990fb851 WatchSource:0}: Error finding container 78f2690cbfcb4b7b1e1ebc02a2252fe7a01232b169ce1e71c8ed3546990fb851: Status 404 returned error can't find the container with id 78f2690cbfcb4b7b1e1ebc02a2252fe7a01232b169ce1e71c8ed3546990fb851 Oct 30 21:34:09.968662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.968536 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:34:09.970922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.970892 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:09.971016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.970957 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:09.971016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:09.970988 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:10.228271 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:10.228184 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" event={"ID":"586eea25-2b06-4402-9c43-ebfe2bbe5caa","Type":"ContainerStarted","Data":"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909"} Oct 30 21:34:10.228271 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:10.228240 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" event={"ID":"586eea25-2b06-4402-9c43-ebfe2bbe5caa","Type":"ContainerStarted","Data":"78f2690cbfcb4b7b1e1ebc02a2252fe7a01232b169ce1e71c8ed3546990fb851"} Oct 30 21:34:10.348653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:10.348589 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podStartSLOduration=1.34857352 podStartE2EDuration="1.34857352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:10.343846332 +0000 UTC m=+1364.326546753" watchObservedRunningTime="2026-10-30 21:34:10.34857352 +0000 UTC m=+1364.331274026" Oct 30 21:34:10.530946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:10.530850 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:11.534169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:11.534129 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:11.534602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:11.534192 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:12.530641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:12.530596 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:12.530853 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:12.530681 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:13.530846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:13.530812 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:13.531339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:13.530870 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:14.530560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:14.530521 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:14.530748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:14.530585 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:15.530841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:15.530802 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:15.531297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:15.530865 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:16.530623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:16.530589 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 21:34:16.530790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:16.530655 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 21:34:17.534744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:17.534715 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:17.535226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:17.535212 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:17.535851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:17.535830 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:34:18.050707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:18.050678 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7"] Oct 30 21:34:18.055453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:18.055429 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1d6130-pr4j7"] Oct 30 21:34:18.168699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:18.168672 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086dd2f7-1cf3-45bc-88fc-ecc447df458d" path="/var/lib/kubelet/pods/086dd2f7-1cf3-45bc-88fc-ecc447df458d/volumes" Oct 30 21:34:24.332738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.332701 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:34:24.339338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.339298 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.396371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.396341 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:34:24.396709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.396679 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.396815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.396728 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.396942 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.396866 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.397009 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.396959 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.402677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.402652 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.402945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.402924 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.403048 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.402995 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.403132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.403106 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.403269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.403253 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqn7r\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504216 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504182 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqn7r\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504288 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504331 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504370 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504420 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504446 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504508 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504567 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504654 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.504836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.504736 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.505563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.505537 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.505868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.505843 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.505938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.505864 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.506173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.506151 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.507658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.507619 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.508171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.508144 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.561380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.561354 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.561552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.561535 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqn7r\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:24.655540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:24.655476 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:25.087501 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:34:25.087471 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8143da2_8b43_40e9_a9a0_43e6cfdd8dbb.slice/crio-f61c8dea40ca0237df96b9b8860795f9ef75e4d906dac41102dcf2fcaeb808e2 WatchSource:0}: Error finding container f61c8dea40ca0237df96b9b8860795f9ef75e4d906dac41102dcf2fcaeb808e2: Status 404 returned error can't find the container with id f61c8dea40ca0237df96b9b8860795f9ef75e4d906dac41102dcf2fcaeb808e2 Oct 30 21:34:25.089151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.089083 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:34:25.090125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.090088 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:25.090190 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.090174 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:25.090259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.090244 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:25.350579 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.350497 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:34:25.354794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.354771 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.357911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.357885 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" event={"ID":"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb","Type":"ContainerStarted","Data":"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f"} Oct 30 21:34:25.358009 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.357933 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" event={"ID":"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb","Type":"ContainerStarted","Data":"f61c8dea40ca0237df96b9b8860795f9ef75e4d906dac41102dcf2fcaeb808e2"} Oct 30 21:34:25.411669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.411636 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:34:25.415669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415643 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.415792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415743 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.415846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415808 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.415846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415835 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.415913 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415899 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.415984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.415970 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.416040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.416028 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.416100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.416088 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.416211 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.416195 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4tmj\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517604 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517802 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517665 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517802 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517693 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517802 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517742 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517815 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517855 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517900 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.517993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.517987 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4tmj\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.518197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.518057 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.518197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.518096 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.518928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.518421 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.518928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.518597 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.518928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.518681 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.519232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.519200 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.521017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.520988 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.521507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.521483 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.574479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.574447 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4tmj\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.574868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.574847 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\") pod \"e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:25.595285 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.595227 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podStartSLOduration=1.5952091130000001 podStartE2EDuration="1.595209113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:25.590010521 +0000 UTC m=+1379.572710933" watchObservedRunningTime="2026-10-30 21:34:25.595209113 +0000 UTC m=+1379.577909526" Oct 30 21:34:25.656076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.656003 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:25.668516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:25.668490 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:26.113958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.113175 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:34:26.118340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.116201 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:26.118340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.116305 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:26.118340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.116364 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:26.412408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.398675 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" event={"ID":"3b8fbb45-2be4-4716-966b-df3222baccbe","Type":"ContainerStarted","Data":"94e02d49cc0a06a731a452e4f8a222d58e058ff7d74ec57f61c246c0bcc9b64e"} Oct 30 21:34:26.412408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.398764 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" event={"ID":"3b8fbb45-2be4-4716-966b-df3222baccbe","Type":"ContainerStarted","Data":"29f89f7062786538363d08d13fd8368f3c31ff293fd41f1f569904425a97c336"} Oct 30 21:34:26.441928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.441898 4921 scope.go:122] "RemoveContainer" containerID="3a58663587fa09e7438b1c57648f0e817d849cc5c57210361f1b65c1aba9f7ba" Oct 30 21:34:26.660357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.658879 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:26.660357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.658930 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:26.669423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:26.669358 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:27.660339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:27.659722 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:27.660339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:27.659783 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:27.670856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:27.670816 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:27.670992 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:27.670881 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:28.662354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:28.661706 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:28.662354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:28.661766 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:28.669103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:28.669079 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:28.669205 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:28.669120 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:29.659758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:29.659718 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:29.660070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:29.660043 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:29.668899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:29.668869 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:29.669344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:29.669297 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:30.050337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.049650 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podStartSLOduration=5.04962925 podStartE2EDuration="5.04962925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:26.547993376 +0000 UTC m=+1380.530693797" watchObservedRunningTime="2026-10-30 21:34:30.04962925 +0000 UTC m=+1384.032329669" Oct 30 21:34:30.066390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.065212 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk"] Oct 30 21:34:30.111581 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.111542 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac356aa-jxrqk"] Oct 30 21:34:30.171386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.171351 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e117bd3a-59b7-4798-9308-4f39c683cff6" path="/var/lib/kubelet/pods/e117bd3a-59b7-4798-9308-4f39c683cff6/volumes" Oct 30 21:34:30.662481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.662448 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:30.662771 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.662741 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:30.669329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.669295 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:30.669712 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:30.669693 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:31.656011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:31.655973 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:34:31.656184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:31.656040 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:34:31.669504 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:31.669469 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:31.669849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:31.669532 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:32.669494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:32.669459 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:34:32.669715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:32.669510 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:34:32.911538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:32.911504 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:32.911983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:32.911962 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:32.912770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:32.912741 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:34:33.910219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:33.910186 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:33.912196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:33.912176 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:33.913786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:33.913767 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:34:36.333339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:36.332427 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:36.490474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:36.490422 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:36.564061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:36.564017 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:36.636748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:36.636648 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:34:36.937571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:36.933413 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.016610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.016557 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.033215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.032423 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.076476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.076362 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Oct 30 21:34:37.228460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.228408 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.339669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.337435 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:34:37.339669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.338108 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:34:37.432308 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.432263 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.492936 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.492842 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.505341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.493548 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:34:37.605491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.605440 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.606128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.606104 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:34:37.610064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.610044 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:34:37.675440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.675384 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.676248 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.676223 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:34:37.936172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.936075 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:37.937115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:37.937089 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:34:38.087341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.042974 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:38.127340 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:34:38.101990 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:34:38.128069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.128041 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:34:38.421471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.244879 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:38.421471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.383468 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:38.453860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.453825 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:34:38.454056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.453887 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:34:38.454120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.454042 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:38.454424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.454402 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:34:38.475205 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.475175 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:34:38.475507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.475492 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:34:38.476367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.476349 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:34:38.480329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.477744 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:34:38.481946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.481928 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:34:38.491508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:38.491487 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:34:44.052001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:44.051965 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5"] Oct 30 21:34:44.101103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:44.101066 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b63bb1-k5tn5"] Oct 30 21:34:44.171765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:44.171730 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9475243-aeb8-4c2b-8fb8-cf4dab7167da" path="/var/lib/kubelet/pods/b9475243-aeb8-4c2b-8fb8-cf4dab7167da/volumes" Oct 30 21:34:55.228497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.228466 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:34:55.233096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.233075 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.315803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.315776 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.315934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.315888 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.315977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.315954 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.315979 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.316021 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316168 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.316136 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316208 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.316187 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslws\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.316347 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.316414 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.316399 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.337611 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.337586 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:34:55.417984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.417946 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418049 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418100 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418140 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418171 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418224 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418262 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslws\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418402 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418471 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418495 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418543 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418843 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.418999 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.418877 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.419043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.419001 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.420547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.420526 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.420800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.420782 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.474438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.474413 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslws\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.474546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.474447 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.548142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.548076 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:55.978576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.978531 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:34:55.978729 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:34:55.978656 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7ee2a4_3c61_4a37_94d0_caa9b55aead2.slice/crio-d988a4382ab744d1aee61959e5f86670c16cfe213f2a43936331b234a2d7d078 WatchSource:0}: Error finding container d988a4382ab744d1aee61959e5f86670c16cfe213f2a43936331b234a2d7d078: Status 404 returned error can't find the container with id d988a4382ab744d1aee61959e5f86670c16cfe213f2a43936331b234a2d7d078 Oct 30 21:34:55.981362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.981304 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:55.981455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.981392 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:55.981455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:55.981421 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:34:56.042393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:56.042368 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" event={"ID":"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2","Type":"ContainerStarted","Data":"d988a4382ab744d1aee61959e5f86670c16cfe213f2a43936331b234a2d7d078"} Oct 30 21:34:57.051244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:57.051209 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" event={"ID":"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2","Type":"ContainerStarted","Data":"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d"} Oct 30 21:34:57.226847 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:57.226784 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podStartSLOduration=2.226767117 podStartE2EDuration="2.226767117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:57.221620762 +0000 UTC m=+1411.204321181" watchObservedRunningTime="2026-10-30 21:34:57.226767117 +0000 UTC m=+1411.209467535" Oct 30 21:34:57.548937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:57.548909 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:34:57.550899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:57.550868 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:34:57.551028 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:57.550924 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:34:58.548478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:58.548441 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:34:58.548863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:58.548507 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:34:59.549281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:59.549243 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:34:59.549887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:34:59.549856 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:35:00.525949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.525908 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc"] Oct 30 21:35:00.537795 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.537768 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.548982 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.548955 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:35:00.549161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.549138 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:35:00.567504 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.567482 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc"] Oct 30 21:35:00.620335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.615584 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:35:00.620335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.615651 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:35:00.621711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.621684 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.622049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.622031 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wtm\" (UniqueName: \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.622195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.622180 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.733022 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.732985 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wtm\" (UniqueName: \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.733193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.733077 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.733193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.733183 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.735556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.735525 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.738161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.738118 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.792128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.792062 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wtm\" (UniqueName: \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\") pod \"maas-api-revoke-sub-981f88fa-cj8sc\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:00.854228 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:00.854196 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:01.264671 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:01.264646 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9469a73a_3eb3_4011_8b25_3da08abc2479.slice/crio-6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd WatchSource:0}: Error finding container 6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd: Status 404 returned error can't find the container with id 6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd Oct 30 21:35:01.265974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:01.265945 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc"] Oct 30 21:35:01.549283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:01.549243 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:35:01.549453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:01.549297 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:35:02.090699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:02.090666 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" event={"ID":"9469a73a-3eb3-4011-8b25-3da08abc2479","Type":"ContainerStarted","Data":"05690b1d60db9a9364fc81ad38d6f1c5f15ea9d2cacefc29b67fa7df69145b97"} Oct 30 21:35:02.091124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:02.090722 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" event={"ID":"9469a73a-3eb3-4011-8b25-3da08abc2479","Type":"ContainerStarted","Data":"6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd"} Oct 30 21:35:02.208957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:02.208893 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" podStartSLOduration=2.2088751110000002 podStartE2EDuration="2.208875111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:02.20544659 +0000 UTC m=+1416.188147010" watchObservedRunningTime="2026-10-30 21:35:02.208875111 +0000 UTC m=+1416.191575530" Oct 30 21:35:02.549227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:02.549197 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:35:02.549373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:02.549254 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 21:35:03.100804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:03.100764 4921 generic.go:356] "Generic (PLEG): container finished" podID="9469a73a-3eb3-4011-8b25-3da08abc2479" containerID="05690b1d60db9a9364fc81ad38d6f1c5f15ea9d2cacefc29b67fa7df69145b97" exitCode=0 Oct 30 21:35:03.101259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:03.101037 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" event={"ID":"9469a73a-3eb3-4011-8b25-3da08abc2479","Type":"ContainerDied","Data":"05690b1d60db9a9364fc81ad38d6f1c5f15ea9d2cacefc29b67fa7df69145b97"} Oct 30 21:35:03.554221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:03.554192 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:35:03.554684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:03.554663 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:35:03.555211 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:03.555195 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:35:04.307149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.307123 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:04.393651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.393573 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\") pod \"9469a73a-3eb3-4011-8b25-3da08abc2479\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " Oct 30 21:35:04.393794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.393705 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\") pod \"9469a73a-3eb3-4011-8b25-3da08abc2479\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " Oct 30 21:35:04.393794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.393757 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\" (UniqueName: \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\") pod \"9469a73a-3eb3-4011-8b25-3da08abc2479\" (UID: \"9469a73a-3eb3-4011-8b25-3da08abc2479\") " Oct 30 21:35:04.394054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.394028 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca" pod "9469a73a-3eb3-4011-8b25-3da08abc2479" (UID: "9469a73a-3eb3-4011-8b25-3da08abc2479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:04.394434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.394392 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:04.395968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.395944 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca" pod "9469a73a-3eb3-4011-8b25-3da08abc2479" (UID: "9469a73a-3eb3-4011-8b25-3da08abc2479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:04.396231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.396211 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm" pod "9469a73a-3eb3-4011-8b25-3da08abc2479" (UID: "9469a73a-3eb3-4011-8b25-3da08abc2479"). InnerVolumeSpecName "kube-api-access-49wtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:04.495925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.495882 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9469a73a-3eb3-4011-8b25-3da08abc2479-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:04.495925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:04.495918 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49wtm\" (UniqueName: \"kubernetes.io/projected/9469a73a-3eb3-4011-8b25-3da08abc2479-kube-api-access-49wtm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:05.063591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.063558 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt"] Oct 30 21:35:05.064221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.064205 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9469a73a-3eb3-4011-8b25-3da08abc2479" containerName="revoke-keys" Oct 30 21:35:05.064264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.064222 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9469a73a-3eb3-4011-8b25-3da08abc2479" containerName="revoke-keys" Oct 30 21:35:05.064368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.064357 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9469a73a-3eb3-4011-8b25-3da08abc2479" containerName="revoke-keys" Oct 30 21:35:05.068078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.068063 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.103584 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.103559 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.103777 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.103756 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bm8\" (UniqueName: \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.103842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.103816 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.120754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.120597 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" Oct 30 21:35:05.120888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.120874 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc" event={"ID":"9469a73a-3eb3-4011-8b25-3da08abc2479","Type":"ContainerDied","Data":"6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd"} Oct 30 21:35:05.120955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.120907 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9dfb2c810875c3df00c9cf4e27fa25cc9d2d64a4c2b3094d2dfc356b2be6fd" Oct 30 21:35:05.126249 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.126228 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt"] Oct 30 21:35:05.205217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.205187 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bm8\" (UniqueName: \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.205365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.205279 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.205438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.205418 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.206279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.206255 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.207612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.207589 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.261089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.261060 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bm8\" (UniqueName: \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\") pod \"maas-api-revoke-sub-c88e6742-c4kbt\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.382281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.382219 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:05.768218 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:05.768188 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt"] Oct 30 21:35:05.768777 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:05.768744 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be3b305_61ee_43f4_82e9_710b2e07628c.slice/crio-f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d WatchSource:0}: Error finding container f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d: Status 404 returned error can't find the container with id f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d Oct 30 21:35:06.130729 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:06.130700 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" event={"ID":"6be3b305-61ee-43f4-82e9-710b2e07628c","Type":"ContainerStarted","Data":"a18c6937e759184c692fcecc347bafff28ac746941cb2410c838e4968f55a740"} Oct 30 21:35:06.130860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:06.130752 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" event={"ID":"6be3b305-61ee-43f4-82e9-710b2e07628c","Type":"ContainerStarted","Data":"f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d"} Oct 30 21:35:06.248196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:06.248149 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" podStartSLOduration=1.248136127 podStartE2EDuration="1.248136127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:06.24360137 +0000 UTC m=+1420.226301786" watchObservedRunningTime="2026-10-30 21:35:06.248136127 +0000 UTC m=+1420.230836545" Oct 30 21:35:07.140250 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:07.140172 4921 generic.go:356] "Generic (PLEG): container finished" podID="6be3b305-61ee-43f4-82e9-710b2e07628c" containerID="a18c6937e759184c692fcecc347bafff28ac746941cb2410c838e4968f55a740" exitCode=0 Oct 30 21:35:07.140612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:07.140258 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" event={"ID":"6be3b305-61ee-43f4-82e9-710b2e07628c","Type":"ContainerDied","Data":"a18c6937e759184c692fcecc347bafff28ac746941cb2410c838e4968f55a740"} Oct 30 21:35:07.540411 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:07.540376 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:35:09.571339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.530917 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:09.571339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.550435 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:09.571339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.550540 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:09.571339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.550597 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 21:35:09.600453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.600410 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:09.616459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.616403 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 21:35:09.641757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.641715 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:09.643075 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.643046 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:09.671419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.671365 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:09.749635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.688670 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:09.857139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:09.855990 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:10.032448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.032403 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:10.035472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.035438 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:10.057616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.057588 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:10.186770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.186679 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\") pod \"6be3b305-61ee-43f4-82e9-710b2e07628c\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " Oct 30 21:35:10.187021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.187001 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\") pod \"6be3b305-61ee-43f4-82e9-710b2e07628c\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " Oct 30 21:35:10.187274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.187258 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\" (UniqueName: \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\") pod \"6be3b305-61ee-43f4-82e9-710b2e07628c\" (UID: \"6be3b305-61ee-43f4-82e9-710b2e07628c\") " Oct 30 21:35:10.190094 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.190065 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca" pod "6be3b305-61ee-43f4-82e9-710b2e07628c" (UID: "6be3b305-61ee-43f4-82e9-710b2e07628c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:10.222545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.216422 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8" pod "6be3b305-61ee-43f4-82e9-710b2e07628c" (UID: "6be3b305-61ee-43f4-82e9-710b2e07628c"). InnerVolumeSpecName "kube-api-access-d8bm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:10.272813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.272775 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca" pod "6be3b305-61ee-43f4-82e9-710b2e07628c" (UID: "6be3b305-61ee-43f4-82e9-710b2e07628c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:10.356344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.324875 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bm8\" (UniqueName: \"kubernetes.io/projected/6be3b305-61ee-43f4-82e9-710b2e07628c-kube-api-access-d8bm8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:10.356624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.356587 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:10.356766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.356750 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be3b305-61ee-43f4-82e9-710b2e07628c-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:10.598362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.569190 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:10.598362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.569623 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.715916 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.716266 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.716556 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.716814 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.717109 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.717170 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.717208 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:10.718346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.717243 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.748933 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.749344 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.750580 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.750872 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.753111 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:10.773431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.760448 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:10.905339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.900465 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" event={"ID":"6be3b305-61ee-43f4-82e9-710b2e07628c","Type":"ContainerDied","Data":"f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d"} Oct 30 21:35:10.909338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.908474 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt" Oct 30 21:35:10.912635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:10.912600 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f281a901ae2d42953766818c979c1e66967b120c17f91962bdbe734b2260937d" Oct 30 21:35:11.057423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.035774 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:11.057423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.036241 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:11.141844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.141445 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 21:35:11.577916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.577495 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:11.608593 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:11.608552 4921 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be3b305_61ee_43f4_82e9_710b2e07628c.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:35:11.769341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.718444 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:11.769341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.722410 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:11.795131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.770831 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:11.795131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.771664 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:11.795131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.772339 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:11.795131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.772600 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:11.795131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:11.775508 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:12.036341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.036247 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.437396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.433723 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.600482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.583232 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.738340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.737933 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.777375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.758734 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:12.790346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.782416 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.790346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.783248 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.790346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.783570 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:12.790346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.783804 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:12.790346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:12.784036 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:13.082557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.082460 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:13.591350 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.583691 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:13.748902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.748859 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:13.763840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.763796 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:13.859546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.859441 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:13.860394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.860361 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:13.869448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.869408 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:13.870736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.870705 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:13.887958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:13.887920 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:14.087448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.087400 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:14.245791 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.245763 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:14.246064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.246052 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:14.246173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.246164 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:14.319955 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.319926 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:14.352334 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.349154 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:14.352808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.352784 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:14.363124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.362360 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:14.363124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.362419 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:14.400992 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:14.400913 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:15.798976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.798943 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974"] Oct 30 21:35:15.799593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.799576 4921 cpu_manager.go:414] "containerMap: removing container" podUID="6be3b305-61ee-43f4-82e9-710b2e07628c" containerName="revoke-keys" Oct 30 21:35:15.799593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.799593 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be3b305-61ee-43f4-82e9-710b2e07628c" containerName="revoke-keys" Oct 30 21:35:15.799717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.799708 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be3b305-61ee-43f4-82e9-710b2e07628c" containerName="revoke-keys" Oct 30 21:35:15.803153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.803133 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:15.855290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.855247 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:35:15.860668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.860638 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974"] Oct 30 21:35:15.903207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.903175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:15.903383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.903210 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2rr\" (UniqueName: \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:15.903473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.903454 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:15.904393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:15.904373 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:35:16.004846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.004815 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.005056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.004910 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.005056 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.004929 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2rr\" (UniqueName: \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.005636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.005613 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.007293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.007269 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.061147 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.061072 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2rr\" (UniqueName: \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\") pod \"maas-api-revoke-sub-a28b059e-gv974\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.119255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.119223 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:16.508099 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:16.508067 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08b4a33_3e00_4034_8f2f_e2e182fe3f76.slice/crio-8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e WatchSource:0}: Error finding container 8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e: Status 404 returned error can't find the container with id 8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e Oct 30 21:35:16.508925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:16.508895 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974"] Oct 30 21:35:17.079701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.079657 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" event={"ID":"b08b4a33-3e00-4034-8f2f-e2e182fe3f76","Type":"ContainerStarted","Data":"7dcc0350b6028bcc03cfb78361cf1884fcf46f640c1416050bd77b3b90f94e55"} Oct 30 21:35:17.080102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.079713 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" event={"ID":"b08b4a33-3e00-4034-8f2f-e2e182fe3f76","Type":"ContainerStarted","Data":"8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e"} Oct 30 21:35:17.200902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.200853 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" podStartSLOduration=2.200836884 podStartE2EDuration="2.200836884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:17.198709265 +0000 UTC m=+1431.181409690" watchObservedRunningTime="2026-10-30 21:35:17.200836884 +0000 UTC m=+1431.183537302" Oct 30 21:35:17.680584 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.680548 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-5c6627"] Oct 30 21:35:17.684718 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.684692 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:17.738337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.738106 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmshp\"" Oct 30 21:35:17.746873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.746835 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-5c6627"] Oct 30 21:35:17.834806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.834702 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlkj\" (UniqueName: \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\") pod \"test-curl-7202-5c6627\" (UID: \"5452e9c2-a044-4c84-b2bc-9e55b8963156\") " pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:17.936067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:17.936034 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlkj\" (UniqueName: \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\") pod \"test-curl-7202-5c6627\" (UID: \"5452e9c2-a044-4c84-b2bc-9e55b8963156\") " pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:18.007219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.006504 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlkj\" (UniqueName: \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\") pod \"test-curl-7202-5c6627\" (UID: \"5452e9c2-a044-4c84-b2bc-9e55b8963156\") " pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:18.091744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.091660 4921 generic.go:356] "Generic (PLEG): container finished" podID="b08b4a33-3e00-4034-8f2f-e2e182fe3f76" containerID="7dcc0350b6028bcc03cfb78361cf1884fcf46f640c1416050bd77b3b90f94e55" exitCode=0 Oct 30 21:35:18.092214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.091749 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" event={"ID":"b08b4a33-3e00-4034-8f2f-e2e182fe3f76","Type":"ContainerDied","Data":"7dcc0350b6028bcc03cfb78361cf1884fcf46f640c1416050bd77b3b90f94e55"} Oct 30 21:35:18.297462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.297431 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:18.686249 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:18.686219 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5452e9c2_a044_4c84_b2bc_9e55b8963156.slice/crio-7e65266ba422194d7a29dabdb523653c467e05a30f12a540ed8b3aee734022cc WatchSource:0}: Error finding container 7e65266ba422194d7a29dabdb523653c467e05a30f12a540ed8b3aee734022cc: Status 404 returned error can't find the container with id 7e65266ba422194d7a29dabdb523653c467e05a30f12a540ed8b3aee734022cc Oct 30 21:35:18.686688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.686661 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-5c6627"] Oct 30 21:35:18.688290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:18.688273 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:35:19.109843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.109799 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-5c6627" event={"ID":"5452e9c2-a044-4c84-b2bc-9e55b8963156","Type":"ContainerStarted","Data":"7e65266ba422194d7a29dabdb523653c467e05a30f12a540ed8b3aee734022cc"} Oct 30 21:35:19.293835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.293809 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:19.354162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.354132 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\") pod \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " Oct 30 21:35:19.354358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.354180 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\") pod \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " Oct 30 21:35:19.354358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.354251 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\" (UniqueName: \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\") pod \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\" (UID: \"b08b4a33-3e00-4034-8f2f-e2e182fe3f76\") " Oct 30 21:35:19.354564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.354539 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca" pod "b08b4a33-3e00-4034-8f2f-e2e182fe3f76" (UID: "b08b4a33-3e00-4034-8f2f-e2e182fe3f76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:19.354698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.354684 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:19.356210 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.356188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca" pod "b08b4a33-3e00-4034-8f2f-e2e182fe3f76" (UID: "b08b4a33-3e00-4034-8f2f-e2e182fe3f76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:19.356291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.356271 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr" pod "b08b4a33-3e00-4034-8f2f-e2e182fe3f76" (UID: "b08b4a33-3e00-4034-8f2f-e2e182fe3f76"). InnerVolumeSpecName "kube-api-access-6h2rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:19.456180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.456139 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2rr\" (UniqueName: \"kubernetes.io/projected/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-kube-api-access-6h2rr\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:19.456180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:19.456180 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08b4a33-3e00-4034-8f2f-e2e182fe3f76-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:20.121191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:20.121150 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" event={"ID":"b08b4a33-3e00-4034-8f2f-e2e182fe3f76","Type":"ContainerDied","Data":"8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e"} Oct 30 21:35:20.121669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:20.121231 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8102b12e85d1fb7a6f4ad745ac5c0ea4f5abc71cf2b9886e2fabfe8d6a381f1e" Oct 30 21:35:20.121669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:20.121286 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974" Oct 30 21:35:22.147779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:22.147742 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-5c6627" event={"ID":"5452e9c2-a044-4c84-b2bc-9e55b8963156","Type":"ContainerStarted","Data":"2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d"} Oct 30 21:35:23.726796 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.726725 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-5c6627" podStartSLOduration=3.926224984 podStartE2EDuration="6.726708364s" totalImagesPullingTime="2.80048338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:22.273432147 +0000 UTC m=+1436.256132565" watchObservedRunningTime="2026-10-30 21:35:23.726708364 +0000 UTC m=+1437.709408782" Oct 30 21:35:23.732408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.732377 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-5c6627"] Oct 30 21:35:23.739438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.739406 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-5c6627"] Oct 30 21:35:23.887291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.887258 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7202-9963b8"] Oct 30 21:35:23.887894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.887880 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b08b4a33-3e00-4034-8f2f-e2e182fe3f76" containerName="revoke-keys" Oct 30 21:35:23.887937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.887895 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08b4a33-3e00-4034-8f2f-e2e182fe3f76" containerName="revoke-keys" Oct 30 21:35:23.887937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.887927 4921 cpu_manager.go:414] "containerMap: removing container" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" containerName="test-curl-7202-5c6627" Oct 30 21:35:23.887937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.887933 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" containerName="test-curl-7202-5c6627" Oct 30 21:35:23.888043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.888034 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08b4a33-3e00-4034-8f2f-e2e182fe3f76" containerName="revoke-keys" Oct 30 21:35:23.888076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.888051 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" containerName="test-curl-7202-5c6627" Oct 30 21:35:23.891615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.891589 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:23.948562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.948535 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-9963b8"] Oct 30 21:35:23.971448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:23.971421 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jq9\" (UniqueName: \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\") pod \"test-curl-7202-9963b8\" (UID: \"3a27968e-1972-4963-9d49-e123881c8398\") " pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:24.072423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.072310 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jq9\" (UniqueName: \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\") pod \"test-curl-7202-9963b8\" (UID: \"3a27968e-1972-4963-9d49-e123881c8398\") " pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:24.128019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.127982 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jq9\" (UniqueName: \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\") pod \"test-curl-7202-9963b8\" (UID: \"3a27968e-1972-4963-9d49-e123881c8398\") " pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:24.178071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.178022 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-5c6627" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" containerName="test-curl-7202-5c6627" containerID="cri-o://2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d" gracePeriod=2 Oct 30 21:35:24.205963 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.205936 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:24.596539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.596508 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7202-9963b8"] Oct 30 21:35:24.596938 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:24.596918 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a27968e_1972_4963_9d49_e123881c8398.slice/crio-c497ed0b409e02998b818acf6069483d8a7702629e68ee83ada57f14b8dc61db WatchSource:0}: Error finding container c497ed0b409e02998b818acf6069483d8a7702629e68ee83ada57f14b8dc61db: Status 404 returned error can't find the container with id c497ed0b409e02998b818acf6069483d8a7702629e68ee83ada57f14b8dc61db Oct 30 21:35:24.832102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.832072 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:35:24.836283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.836264 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.934937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.932707 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:35:24.985829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.985801 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4s9d\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.985990 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.985872 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.985990 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.985956 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.985998 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.986092 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.986146 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986243 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.986230 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.986264 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:24.986311 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:24.986291 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087095 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087167 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087218 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087261 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087301 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087426 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4s9d\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087514 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087549 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087592 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087655 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087769 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087727 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087826 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087790 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.087976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.087954 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.088034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.088019 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.089760 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.089734 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.089976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.089960 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.160219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.160182 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.160390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.160287 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4s9d\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\") pod \"e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.187476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.187392 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-9963b8" event={"ID":"3a27968e-1972-4963-9d49-e123881c8398","Type":"ContainerStarted","Data":"7daa2bd4e81b2e8dbc43e432af54942efc2b006e5c640ec70873c8412d560891"} Oct 30 21:35:25.187476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.187446 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7202-9963b8" event={"ID":"3a27968e-1972-4963-9d49-e123881c8398","Type":"ContainerStarted","Data":"c497ed0b409e02998b818acf6069483d8a7702629e68ee83ada57f14b8dc61db"} Oct 30 21:35:25.255865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.255829 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:35:25.261512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.261490 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.289180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.289144 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:35:25.293302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.293282 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.303492 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.303436 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7202-9963b8" podStartSLOduration=2.3034182469999998 podStartE2EDuration="2.303418247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:25.301225773 +0000 UTC m=+1439.283926190" watchObservedRunningTime="2026-10-30 21:35:25.303418247 +0000 UTC m=+1439.286118666" Oct 30 21:35:25.326410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.326379 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:35:25.355364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.355307 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:35:25.391128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391093 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391332 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391144 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391332 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391284 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391359 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391403 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391514 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391444 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391514 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391501 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391526 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391638 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391677 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391704 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391802 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.391893 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391879 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391915 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whlmg\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.391982 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.391936 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.392049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.392030 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.392107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.392074 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctd4\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.392146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.392121 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.449480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.449448 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:25.493099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493062 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493113 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493154 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493185 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493226 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493269 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493327 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493365 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493440 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493487 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493511 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493530 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493576 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493624 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493645 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493709 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whlmg\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493746 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.493843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493807 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493848 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctd4\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493889 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.493948 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.494012 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.494383 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.495087 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496005 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496181 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496510 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496598 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496718 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.497699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.496853 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.498285 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.498231 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.498533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.498500 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.501042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.501016 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:35:25.508076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.506522 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.558524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.558483 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.558730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.558705 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.560496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.560472 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctd4\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.560800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.560776 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whlmg\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\") pod \"e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.571784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.571752 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:35:25.578583 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.578556 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:25.594916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.594890 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595052 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595032 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595092 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595167 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcn8q\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595274 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595349 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595418 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595475 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.595538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.595522 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.608124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.608095 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:25.696925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.696889 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.697074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.697014 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.697074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.697056 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.697200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.697115 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcn8q\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.697973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.697936 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.698093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.698004 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.698093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.698071 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.698206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.698108 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.698206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.698154 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.698857 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.698788 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.699194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.699172 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.699525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.699480 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.699921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.699862 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.701685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.701660 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.702097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.702073 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.703389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.703370 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.772401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.772363 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcn8q\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.773810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.773782 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\") pod \"e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.829877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.829848 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:25.890835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.890535 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:35:25.900438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.900397 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:25.900544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.900493 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:25.900544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.900539 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:25.982380 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:25.982349 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b71d9c_4cb4_4653_bf8f_802d4ad24e79.slice/crio-a5e4e60816718062fd10125af816aa612c5e1e3c665f729966d682df8d9dd2c5 WatchSource:0}: Error finding container a5e4e60816718062fd10125af816aa612c5e1e3c665f729966d682df8d9dd2c5: Status 404 returned error can't find the container with id a5e4e60816718062fd10125af816aa612c5e1e3c665f729966d682df8d9dd2c5 Oct 30 21:35:25.982493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.982454 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:35:25.985247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.985208 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:25.985363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.985297 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:25.985413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:25.985368 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.031544 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:26.031247 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa26ecea_8202_447f_a30f_74a73ec12f0c.slice/crio-feabf35bb1e2ffa53586fc8ded884ea106f6fb292f1544bb0ac5d2b524591e0e WatchSource:0}: Error finding container feabf35bb1e2ffa53586fc8ded884ea106f6fb292f1544bb0ac5d2b524591e0e: Status 404 returned error can't find the container with id feabf35bb1e2ffa53586fc8ded884ea106f6fb292f1544bb0ac5d2b524591e0e Oct 30 21:35:26.033335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.033290 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:35:26.034409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.034376 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.034494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.034438 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.034494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.034465 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.210552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.210519 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" event={"ID":"aa26ecea-8202-447f-a30f-74a73ec12f0c","Type":"ContainerStarted","Data":"faa2111eecc9e4c3ef9a29c0d465d997a120d23eb52c928407c30c3c384263f0"} Oct 30 21:35:26.210683 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.210585 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" event={"ID":"aa26ecea-8202-447f-a30f-74a73ec12f0c","Type":"ContainerStarted","Data":"feabf35bb1e2ffa53586fc8ded884ea106f6fb292f1544bb0ac5d2b524591e0e"} Oct 30 21:35:26.213140 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.212625 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" event={"ID":"77b71d9c-4cb4-4653-bf8f-802d4ad24e79","Type":"ContainerStarted","Data":"beab3a31d27a8587f2b8fbe0e49f695bc4b1942fed0edd469616fa56c2503858"} Oct 30 21:35:26.213140 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.212668 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" event={"ID":"77b71d9c-4cb4-4653-bf8f-802d4ad24e79","Type":"ContainerStarted","Data":"a5e4e60816718062fd10125af816aa612c5e1e3c665f729966d682df8d9dd2c5"} Oct 30 21:35:26.215764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.215688 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" event={"ID":"6bc2134f-56c5-49af-a0dc-97dc539d20e2","Type":"ContainerStarted","Data":"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623"} Oct 30 21:35:26.215764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.215726 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" event={"ID":"6bc2134f-56c5-49af-a0dc-97dc539d20e2","Type":"ContainerStarted","Data":"f114938bd240e33c6f6574ce3246a1d83aea1e1c508c9229a4bda78f19c31042"} Oct 30 21:35:26.222677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.222655 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:35:26.255331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.254264 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.255331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.254386 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.255331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.254436 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:26.333137 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.329874 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podStartSLOduration=1.3298547250000001 podStartE2EDuration="1.329854725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.328453546 +0000 UTC m=+1440.311153964" watchObservedRunningTime="2026-10-30 21:35:26.329854725 +0000 UTC m=+1440.312555145" Oct 30 21:35:26.456373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.453993 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:26.456373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.454143 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:26.456373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.454188 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:26.541666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.493331 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podStartSLOduration=2.493296154 podStartE2EDuration="2.493296154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.484549846 +0000 UTC m=+1440.467250278" watchObservedRunningTime="2026-10-30 21:35:26.493296154 +0000 UTC m=+1440.475996571" Oct 30 21:35:26.541666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.493374 4921 scope.go:122] "RemoveContainer" containerID="cb9b992318f0cc46a71a545997389f4c79da29359d532ce939c69a247e908e9f" Oct 30 21:35:26.541666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.520041 4921 scope.go:122] "RemoveContainer" containerID="f868fd8bd71f02169b48616bdf7c5ae87ed91b0ce65706cfb3f69c2ce2d337f9" Oct 30 21:35:26.560055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.559992 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:26.578722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.578695 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:26.612368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.607017 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podStartSLOduration=1.6069971060000001 podStartE2EDuration="1.606997106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.605706943 +0000 UTC m=+1440.588407361" watchObservedRunningTime="2026-10-30 21:35:26.606997106 +0000 UTC m=+1440.589697524" Oct 30 21:35:26.616405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.612653 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:26.719345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.717267 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\" (UniqueName: \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\") pod \"5452e9c2-a044-4c84-b2bc-9e55b8963156\" (UID: \"5452e9c2-a044-4c84-b2bc-9e55b8963156\") " Oct 30 21:35:26.724833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.724798 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj" pod "5452e9c2-a044-4c84-b2bc-9e55b8963156" (UID: "5452e9c2-a044-4c84-b2bc-9e55b8963156"). InnerVolumeSpecName "kube-api-access-cdlkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:26.838032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:26.837938 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlkj\" (UniqueName: \"kubernetes.io/projected/5452e9c2-a044-4c84-b2bc-9e55b8963156-kube-api-access-cdlkj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:27.160389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.141296 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7202-9963b8"] Oct 30 21:35:27.514367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.513987 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7202-9963b8"] Oct 30 21:35:27.567340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.566673 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:27.567340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.566735 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:27.611339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.611208 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:27.629345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.626078 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:27.653342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.652347 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:27.653342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.652408 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:27.806542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.806437 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:27.847343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.835754 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" event={"ID":"c10258cf-0d7a-47f2-a2df-4276659beb18","Type":"ContainerStarted","Data":"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812"} Oct 30 21:35:27.967341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.960482 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" event={"ID":"c10258cf-0d7a-47f2-a2df-4276659beb18","Type":"ContainerStarted","Data":"7af0f062f9ea3ee241343a77ef27b6f8d94ea313515a6d8a2147b13e035aea1b"} Oct 30 21:35:27.967341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.960603 4921 scope.go:122] "RemoveContainer" containerID="2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d" Oct 30 21:35:27.967341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.871896 4921 generic.go:356] "Generic (PLEG): container finished" podID="5452e9c2-a044-4c84-b2bc-9e55b8963156" containerID="2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d" exitCode=137 Oct 30 21:35:27.967341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.887860 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:27.996822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:27.977032 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7202-9963b8" podUID="3a27968e-1972-4963-9d49-e123881c8398" containerName="test-curl-7202-9963b8" containerID="cri-o://7daa2bd4e81b2e8dbc43e432af54942efc2b006e5c640ec70873c8412d560891" gracePeriod=2 Oct 30 21:35:28.087412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.068479 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podStartSLOduration=3.068456083 podStartE2EDuration="3.068456083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:28.066420515 +0000 UTC m=+1442.049120935" watchObservedRunningTime="2026-10-30 21:35:28.068456083 +0000 UTC m=+1442.051156501" Oct 30 21:35:28.132634 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:28.132574 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-9963b8\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="3a27968e-1972-4963-9d49-e123881c8398" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:28.482348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.457411 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:28.482348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.457469 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:28.589434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.589383 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:28.621088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.620997 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:28.621366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.621332 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:28.688593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.688555 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:28.688882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.688852 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:28.796380 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:28.795614 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-5c6627\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" pod="opendatahub/test-curl-7202-5c6627" Oct 30 21:35:28.825340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.821645 4921 scope.go:122] "RemoveContainer" containerID="2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d" Oct 30 21:35:28.825340 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:28.822131 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d\": container with ID starting with 2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d not found: ID does not exist" containerID="2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d" Oct 30 21:35:28.825340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.822166 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d"} err="failed to get container status \"2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d\": rpc error: code = NotFound desc = could not find container \"2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d\": container with ID starting with 2ce07b5a0f568177a0b6b7445de8cfb8e6d2b41b2c5a6910fe53c70854aa6d8d not found: ID does not exist" Oct 30 21:35:28.850338 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:28.848725 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7202-9963b8\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="3a27968e-1972-4963-9d49-e123881c8398" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:28.850338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.849415 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5452e9c2-a044-4c84-b2bc-9e55b8963156" path="/var/lib/kubelet/pods/5452e9c2-a044-4c84-b2bc-9e55b8963156/volumes" Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.851124 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.852047 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.861761 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.861819 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.866528 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:28.876103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.869210 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:28.879006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.878980 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:28.887733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.887705 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:28.892661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.892642 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:28.892841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.892828 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:35:28.919148 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.919123 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3a27968e-1972-4963-9d49-e123881c8398" containerName="test-curl-7202-9963b8" Oct 30 21:35:28.919772 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.919739 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a27968e-1972-4963-9d49-e123881c8398" containerName="test-curl-7202-9963b8" Oct 30 21:35:28.923664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:28.923627 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a27968e-1972-4963-9d49-e123881c8398" containerName="test-curl-7202-9963b8" Oct 30 21:35:29.061622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.041387 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.061622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.056767 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.070599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.070570 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:29.152398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.120572 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:35:29.266783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.217858 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6f787654-openshift-default-dockercfg-ljggn\"" Oct 30 21:35:29.349563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.349530 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.349944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.349923 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.350114 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.350099 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.350251 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.350237 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.350637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.350618 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpf2\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.363361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.351126 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.365630 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.365604 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.365897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.365878 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.379341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.366202 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.530344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.526458 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:29.545423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.534889 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:29.565431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.552457 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.565431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.552552 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded" Oct 30 21:35:29.597253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.597217 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.597707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.597686 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.597971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.597952 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.598264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.598196 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.598459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.598440 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.598588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.598573 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.598794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.598779 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpf2\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.598996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.598979 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.599115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.599102 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.642681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.635101 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.800254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.643263 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.800721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.643824 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.800877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.644055 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.800966 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.644702 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.644926 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.819616 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.878428 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.846970 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.878711 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.860416 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.860385 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.860478 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.878907 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.862391 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.878949 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.820111 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.879074 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:29.879399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.879110 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:29.932365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.882761 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:29.932365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:29.909433 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:30.005395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.005359 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpf2\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:30.043819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.012878 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:30.046670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.036062 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 30 21:35:30.046862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.043659 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:30.047638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.047616 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:30.143539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.143433 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:30.229718 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.217663 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:30.252346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.232869 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\") pod \"e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:30.345248 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.345212 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:30.474538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.452054 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:30.474538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.452113 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:30.711420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.687791 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:30.711420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.687846 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:30.711420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.687922 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:30.711420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.687943 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:30.766364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.753413 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:30.766364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.753779 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:35:31.012401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:30.983690 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.013533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.013500 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.013979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.003202 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:31.037340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.016503 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:31.037340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.033674 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:31.037340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.033726 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:31.037340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.012472 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:31.037340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.003146 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.038102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.003469 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:31.039549 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.012420 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:31.054339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.047121 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.054339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.051391 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.144366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.114498 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b25d536b-dockercfg-b99n2\"" Oct 30 21:35:31.144366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.114584 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b25d536b\"" Oct 30 21:35:31.175353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.168780 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:35:31.175353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.169286 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.175353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.169619 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:31.299424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.272470 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.299775 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.299743 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqft\" (UniqueName: \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.449873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.449834 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.450160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.450137 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqft\" (UniqueName: \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.487349 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.451785 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.543340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.529066 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:31.543340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.529125 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:31.606398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.596071 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqft\" (UniqueName: \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\") pod \"payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:31.606398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.596185 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:31.606398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.596226 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:31.802733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.802697 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:31.831340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.814291 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:31.833295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.818127 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:31.833550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.833521 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:31.834050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.834008 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:31.834200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.834174 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:31.884409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:31.884366 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:32.038090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.037954 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.047341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.043415 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.047341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.043759 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.047341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.044025 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.047341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.044372 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:32.088385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.060444 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:32.194598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.194489 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.223343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.198762 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.423171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.236052 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.478452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.478401 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:32.530396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.487189 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:32.530396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.487246 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:32.581856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.581819 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:32.751971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.751870 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:32.752211 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.623352 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:32.756413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.756373 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:32.827495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.827431 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:32.946525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.946258 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:32.946525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:32.946336 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:33.054756 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.054694 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.068340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.060621 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.099371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.068590 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:33.110349 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.068668 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:33.110349 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.072391 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.295392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.291418 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.374340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.370374 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:33.374340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.370507 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:33.374340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.370557 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:33.430260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.429406 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:33.430260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.429406 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.513713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.473079 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:33.513713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.473130 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:33.936458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.932688 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:33.936458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.932735 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:33.936458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.932802 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:33.936458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.932821 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:33.936458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.933622 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:33.997725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.959535 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:33.997725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:33.959605 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:34.081339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.078837 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:34.326606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.092663 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:34.327447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.112340 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:34.329568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.112406 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:34.378241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.169473 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:34.378241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.323196 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:34.449625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.440246 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:34.594395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.540570 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:34.594395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.540635 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:34.610797 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.601837 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:34.896444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.896408 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:34.896717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.896688 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:34.916512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.916475 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:34.916778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.916751 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:34.940659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.940622 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:34.945445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.945396 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:34.991446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:34.970141 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:35.147557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.107418 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:35.169215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.168329 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:35.189907 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.188212 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerStarted","Data":"4ada47b73934c6e6a4242d93f41545609689541b25b8e03c6662024541fa2061"} Oct 30 21:35:35.198333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.194385 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:35.224590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.224549 4921 generic.go:356] "Generic (PLEG): container finished" podID="3a27968e-1972-4963-9d49-e123881c8398" containerID="7daa2bd4e81b2e8dbc43e432af54942efc2b006e5c640ec70873c8412d560891" exitCode=137 Oct 30 21:35:35.225184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.225166 4921 scope.go:122] "RemoveContainer" containerID="7daa2bd4e81b2e8dbc43e432af54942efc2b006e5c640ec70873c8412d560891" Oct 30 21:35:35.225543 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.225527 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7202-9963b8" Oct 30 21:35:35.281676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.281646 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\" (UniqueName: \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\") pod \"3a27968e-1972-4963-9d49-e123881c8398\" (UID: \"3a27968e-1972-4963-9d49-e123881c8398\") " Oct 30 21:35:35.292561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.292518 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9" pod "3a27968e-1972-4963-9d49-e123881c8398" (UID: "3a27968e-1972-4963-9d49-e123881c8398"). InnerVolumeSpecName "kube-api-access-n7jq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:35.325564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.325525 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.326771 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.326738 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.334341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.327795 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.339410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.339381 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.356888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.352909 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.359594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.359557 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:35.387487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.387454 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jq9\" (UniqueName: \"kubernetes.io/projected/3a27968e-1972-4963-9d49-e123881c8398-kube-api-access-n7jq9\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:35.424601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.417340 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:35.483417 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.451935 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:35:35.484755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.484504 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:35.484755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.484583 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:35.485049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.485023 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:35:35.485196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.485172 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:35:35.495460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.495196 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:35.573357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.538361 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:35.575616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.575585 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:35.590336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.582067 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:35.590336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.582137 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:35.592460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.592353 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:35:35.625357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.622911 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 21:35:35.625357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.622965 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 21:35:35.664368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.661557 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:35.831377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.831342 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:35:35.933517 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:35.930905 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:35:36.419633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.414183 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a27968e-1972-4963-9d49-e123881c8398" path="/var/lib/kubelet/pods/3a27968e-1972-4963-9d49-e123881c8398/volumes" Oct 30 21:35:36.420083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.419932 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:36.420083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.419985 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerStarted","Data":"dba0201b200ae739ccaa7e8c8dc53a3ba3a5ef8aaf70d6189d1b643743499cc5"} Oct 30 21:35:36.420083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.420014 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:36.420083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.420053 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" event={"ID":"426afed3-ad13-4c86-a441-ea9babcbcc00","Type":"ContainerStarted","Data":"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c"} Oct 30 21:35:36.420083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.420070 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" event={"ID":"426afed3-ad13-4c86-a441-ea9babcbcc00","Type":"ContainerStarted","Data":"bb3ed60a2bf9d1770bbe2e4bdeb437c7217068e8cec2d59494c5124f127fcd00"} Oct 30 21:35:36.420374 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.420355 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:36.561935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.561584 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:36.562273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.562251 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:36.565770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.565752 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:35:36.580697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.580668 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:35:36.580949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.580906 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 21:35:36.758959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.758899 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" podStartSLOduration=6.758867733 podStartE2EDuration="6.758867733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:36.748466056 +0000 UTC m=+1450.731166475" watchObservedRunningTime="2026-10-30 21:35:36.758867733 +0000 UTC m=+1450.741568151" Oct 30 21:35:36.912682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:36.912619 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podStartSLOduration=8.912592837 podStartE2EDuration="8.912592837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:36.866463325 +0000 UTC m=+1450.849163754" watchObservedRunningTime="2026-10-30 21:35:36.912592837 +0000 UTC m=+1450.895293256" Oct 30 21:35:37.432570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.432300 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:37.433222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.433122 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:35:37.492363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.491143 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:37.492571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.492549 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:37.493266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.493246 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:35:37.517387 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.517355 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:37.518467 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.518444 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:37.522011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.521985 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:35:37.544846 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:37.544815 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:35:37.596143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.596110 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:37.601370 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.597663 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:37.603785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:37.603758 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:35:38.858405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.858368 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:35:38.897688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.897663 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:38.916393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.916361 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:35:38.950195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.950158 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1d5266a5-dockercfg-ngzzt\"" Oct 30 21:35:38.950362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.950247 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1d5266a5\"" Oct 30 21:35:38.959166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.959137 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v22bg\" (UniqueName: \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:38.959355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:38.959284 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.060962 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.060928 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22bg\" (UniqueName: \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.061127 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.061080 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.061740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.061720 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.117874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.117814 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22bg\" (UniqueName: \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\") pod \"payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.234983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.234957 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:39.606292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.606261 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:35:39.612033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.612008 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.620393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.620370 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:35:39.624539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.624522 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.627395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.627361 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:35:39.663089 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:39.663065 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c8c8f2_1e0e_4e6d_ab82_490b77dfcad8.slice/crio-f31d081024449f8f2010a09ae6a6e1a197c7e14a09276a2f52daa4aae6b3373e WatchSource:0}: Error finding container f31d081024449f8f2010a09ae6a6e1a197c7e14a09276a2f52daa4aae6b3373e: Status 404 returned error can't find the container with id f31d081024449f8f2010a09ae6a6e1a197c7e14a09276a2f52daa4aae6b3373e Oct 30 21:35:39.669835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.669814 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.669903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.669859 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.669903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.669886 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670009 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.669985 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670058 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670027 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670137 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670118 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670151 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670261 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670298 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670349 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr4q\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670375 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670439 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670476 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670577 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.670645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670609 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670737 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670723 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670792 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.670860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.670818 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7g9\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.717592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.713472 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:35:39.753365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.753336 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:35:39.771775 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.771748 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.771881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.771786 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.771881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.771816 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.771881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.771838 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.771930 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772002 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772025 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7g9\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772116 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772176 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772212 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772227 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772240 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772231 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772270 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772309 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772373 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772470 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772511 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772534 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr4q\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772544 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772557 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.772985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772915 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.773042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.772998 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.773898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.773675 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.773898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.773719 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.774055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.773961 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.774260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.774225 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.774465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.774445 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.775176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.775149 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.775347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.775295 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.776496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.776472 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.776890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.776871 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.828673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.828645 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7g9\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.828801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.828778 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.833101 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.833080 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr4q\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.833406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.833386 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\") pod \"e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:39.942931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.942590 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:39.944939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:39.944915 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:40.394739 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:40.394710 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25a12b3_893c_43af_9e24_ff2e3f4b84c8.slice/crio-ce900d6c901244245fab38c72bbecb95305945bb8ff7fcf83f6b9022a3c32969 WatchSource:0}: Error finding container ce900d6c901244245fab38c72bbecb95305945bb8ff7fcf83f6b9022a3c32969: Status 404 returned error can't find the container with id ce900d6c901244245fab38c72bbecb95305945bb8ff7fcf83f6b9022a3c32969 Oct 30 21:35:40.395409 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.395386 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:35:40.395899 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:40.395875 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95637a3_5393_46bc_8cfc_9c30ee489828.slice/crio-d5b0a88d1338a722e85cb613a6dea5de02324de09d356874731e328d72423cbc WatchSource:0}: Error finding container d5b0a88d1338a722e85cb613a6dea5de02324de09d356874731e328d72423cbc: Status 404 returned error can't find the container with id d5b0a88d1338a722e85cb613a6dea5de02324de09d356874731e328d72423cbc Oct 30 21:35:40.397543 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.397503 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.397636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.397581 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.397636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.397621 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.398743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.398708 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.398837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.398783 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.398837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.398821 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:35:40.401051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.401030 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:35:40.461203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.461179 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" event={"ID":"b25a12b3-893c-43af-9e24-ff2e3f4b84c8","Type":"ContainerStarted","Data":"ce900d6c901244245fab38c72bbecb95305945bb8ff7fcf83f6b9022a3c32969"} Oct 30 21:35:40.462461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.462438 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" event={"ID":"a95637a3-5393-46bc-8cfc-9c30ee489828","Type":"ContainerStarted","Data":"d5b0a88d1338a722e85cb613a6dea5de02324de09d356874731e328d72423cbc"} Oct 30 21:35:40.463880 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.463861 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" event={"ID":"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8","Type":"ContainerStarted","Data":"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b"} Oct 30 21:35:40.463973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.463899 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" event={"ID":"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8","Type":"ContainerStarted","Data":"f31d081024449f8f2010a09ae6a6e1a197c7e14a09276a2f52daa4aae6b3373e"} Oct 30 21:35:40.463973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.463920 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:40.585337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:40.583413 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" podStartSLOduration=2.5833802439999998 podStartE2EDuration="2.583380244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:40.579871565 +0000 UTC m=+1454.562571985" watchObservedRunningTime="2026-10-30 21:35:40.583380244 +0000 UTC m=+1454.566080662" Oct 30 21:35:41.475747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.475707 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" event={"ID":"b25a12b3-893c-43af-9e24-ff2e3f4b84c8","Type":"ContainerStarted","Data":"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2"} Oct 30 21:35:41.478719 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.478692 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" event={"ID":"a95637a3-5393-46bc-8cfc-9c30ee489828","Type":"ContainerStarted","Data":"80c9f3c7b7ffac08ae619873c3c74619adf56a1557bd870774a5d3b507ca5622"} Oct 30 21:35:41.738396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.738064 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podStartSLOduration=2.738043595 podStartE2EDuration="2.738043595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:41.662619415 +0000 UTC m=+1455.645319832" watchObservedRunningTime="2026-10-30 21:35:41.738043595 +0000 UTC m=+1455.720744014" Oct 30 21:35:41.793117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.793068 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podStartSLOduration=2.7930530129999998 podStartE2EDuration="2.793053013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:41.790577431 +0000 UTC m=+1455.773277849" watchObservedRunningTime="2026-10-30 21:35:41.793053013 +0000 UTC m=+1455.775753430" Oct 30 21:35:41.964605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.964561 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:41.966343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.966298 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:41.966522 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.966438 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:41.966714 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.966681 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:41.967083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.966456 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:41.967246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:41.967217 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:42.949337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:42.949260 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:42.949816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:42.949341 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:42.956340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:42.953615 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:42.956340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:42.953674 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:43.945697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:43.945659 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:44.170341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.169475 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:44.170341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:43.947202 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:44.170341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.169887 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:44.951063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.951025 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:44.951407 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.951371 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:44.951937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.951914 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:44.952082 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:44.952063 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:45.947339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:45.946396 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:45.947339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:45.946455 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:45.947339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:45.946396 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:45.947339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:45.946849 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:46.943246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:46.943209 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:35:46.943456 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:46.943278 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 21:35:46.945623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:46.945596 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:35:46.945738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:46.945649 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 21:35:47.434431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:47.434399 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:35:48.344252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.343760 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:48.351680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.351654 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:48.359568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.354819 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:35:48.359568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.358599 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:48.361108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.360129 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:48.362994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:48.361275 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:35:51.019469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.018402 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.040922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.040883 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.041487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.041448 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:51.042110 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.041659 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:51.049087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.048400 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.056868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.056825 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 21:35:51.076638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.076596 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.106395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.106348 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.312394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.302392 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:51.519339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.518501 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:35:51.952353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:51.932400 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:52.030340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.029547 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:52.030810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.030397 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.059459 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.059845 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.060161 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.060223 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.091006 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.091281 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.091931 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.091990 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.103671 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.103990 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.120931 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:52.152355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.121305 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:52.324424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.312403 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:52.435341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.395899 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:52.435972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.425436 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 21:35:52.997829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:52.997781 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:53.025988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.024819 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:53.270391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.261055 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:53.270391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.262209 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:53.309378 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.280979 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:53.369723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.369677 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:53.400687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.400635 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:53.482343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.478500 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:53.558394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.518748 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:53.815398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.743986 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:53.815398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.763827 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:35:53.815398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:53.763889 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:54.061342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.007617 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:54.410842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.334670 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:54.500341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.342410 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:54.581342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.505386 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:54.581342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.377421 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:54.581342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.406296 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:54.581342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.342469 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:54.581342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.576621 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:54.745453 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:54.744254 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.035042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.034945 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.513353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.511290 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:55.513353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.512952 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:55.513914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.513630 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.513914 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.513878 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:55.557339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.514102 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:55.557877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.521378 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:55.588184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.588148 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.608623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.588100 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.746476 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.746426 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:55.945597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:55.945478 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:56.163345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.150384 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:35:56.625637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.625588 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:35:56.888820 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.663762 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:56.994877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.663805 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:57.157257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.692794 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:57.168411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.692849 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:35:57.183805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.837684 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 21:35:57.243695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:57.243648 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:57.243991 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:57.031009 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:57.255622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:56.979194 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:35:57.306987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:57.306932 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:57.643190 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:35:57.643153 4921 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.434s" Oct 30 21:35:57.683126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:57.673565 4921 trace.go:236] Trace[1516338728]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6c776cd74c-bk6kw" (30-Oct-2026 21:35:56.628) (total time: 1044ms): Oct 30 21:35:57.683126 ip-10-0-137-220 kubenswrapper[4921]: Trace[1516338728]: [1.044886913s] [1.044886913s] END Oct 30 21:35:57.907944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:57.907844 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:58.003978 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.003788 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:35:58.172393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.169285 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 30 21:35:58.182337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.173951 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:35:58.185300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.185270 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:58.208345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.198812 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:58.253346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.251497 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:35:58.267588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.267552 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:35:58.311841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.311813 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:35:58.313419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.313388 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:58.315380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.315354 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:35:58.481342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.478013 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:35:58.481342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.479767 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:35:58.661064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.661036 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:35:58.727355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.727307 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:35:58.748079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.748004 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:35:58.778932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.775743 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:35:58.778932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:58.775802 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:35:59.070695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.070622 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:35:59.078275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.078255 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.145234 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.145202 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:35:59.175199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175163 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175311 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh4w\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175515 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175498 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175579 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175652 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175695 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.175746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.175728 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.208776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.208751 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:35:59.208776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.208764 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6b27n\"" Oct 30 21:35:59.208776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.208753 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:35:59.208991 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.208754 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:35:59.277236 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277195 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277288 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277350 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277440 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277380 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277501 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277677 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277641 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh4w\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.277790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.277769 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.279480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.279452 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.281340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.281275 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.281900 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.281877 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.282664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.282642 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.282774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.282687 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.336528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.336460 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.336801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.336781 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh4w\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x6fgf\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.392249 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.392220 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:35:59.833750 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:35:59.833716 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:35:59.834298 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:35:59.834275 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12540083_1aa3_49d1_9c19_41ff86e481a8.slice/crio-88f8c16fc6c5046683619f264413c6770174685f94c02d99d506620d87f7abd0 WatchSource:0}: Error finding container 88f8c16fc6c5046683619f264413c6770174685f94c02d99d506620d87f7abd0: Status 404 returned error can't find the container with id 88f8c16fc6c5046683619f264413c6770174685f94c02d99d506620d87f7abd0 Oct 30 21:36:00.805057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:00.805020 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" event={"ID":"12540083-1aa3-49d1-9c19-41ff86e481a8","Type":"ContainerStarted","Data":"88f8c16fc6c5046683619f264413c6770174685f94c02d99d506620d87f7abd0"} Oct 30 21:36:04.334390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:04.334306 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:04.334694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:04.334435 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:04.868622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:04.868579 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" event={"ID":"12540083-1aa3-49d1-9c19-41ff86e481a8","Type":"ContainerStarted","Data":"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6"} Oct 30 21:36:04.868818 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:04.868801 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:36:05.088826 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:05.088772 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" podStartSLOduration=1.592412545 podStartE2EDuration="6.088757742s" totalImagesPullingTime="4.496345197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:05.033487159 +0000 UTC m=+1479.016187577" watchObservedRunningTime="2026-10-30 21:36:05.088757742 +0000 UTC m=+1479.071458161" Oct 30 21:36:05.882388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:05.882035 4921 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-x6fgf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:36:05.882388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:05.882092 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:36:07.541677 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:36:07.541587 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:36:08.880691 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:08.880662 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:36:19.541449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.541412 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:36:19.546427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.546406 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.589454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.589427 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:36:19.593660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.593643 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.649929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.649892 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:36:19.660623 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.660503 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:36:19.694125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694089 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694130 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694156 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694272 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694379 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694516 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694407 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694529 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694590 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw2h\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694624 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694654 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694715 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694813 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.694878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694850 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694880 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.694981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.694966 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.695084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.695035 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.695140 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.695107 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.695195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.695178 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz8z\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796379 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796432 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796469 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.796643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796500 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.796643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796540 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796565 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.796851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796734 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796808 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz8z\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796871 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796907 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.796958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796939 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796942 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.796998 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797005 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797104 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797144 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797190 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797218 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797251 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797264 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw2h\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797273 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797301 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797353 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.797783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797476 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797725 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.797783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797769 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.798023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.797980 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.798137 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.798057 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.799746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.799704 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.799946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.799926 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.799997 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.799971 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.800363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.800346 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.853798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.853768 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz8z\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.853908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.853834 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw2h\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.854020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.854002 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.854057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.854005 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\") pod \"e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:19.863012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.862992 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:19.909799 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:19.909626 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:20.294792 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:36:20.294758 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade6ff81_281b_4e00_80d6_d6026fa257df.slice/crio-1b25bfcb1dc6ff1ee8ec1467d2a8eb618d67e51d8acf4d9983bf1107c6130599 WatchSource:0}: Error finding container 1b25bfcb1dc6ff1ee8ec1467d2a8eb618d67e51d8acf4d9983bf1107c6130599: Status 404 returned error can't find the container with id 1b25bfcb1dc6ff1ee8ec1467d2a8eb618d67e51d8acf4d9983bf1107c6130599 Oct 30 21:36:20.295450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.295413 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:36:20.297088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.297057 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.297171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.297115 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.297171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.297143 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.354549 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:36:20.354517 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd9dea5_a48f_404b_b888_809e9ac342b3.slice/crio-bb45dcad6a3b91cb28ebede5ee0eb1ab9ce4da766129e95710a4642501d37527 WatchSource:0}: Error finding container bb45dcad6a3b91cb28ebede5ee0eb1ab9ce4da766129e95710a4642501d37527: Status 404 returned error can't find the container with id bb45dcad6a3b91cb28ebede5ee0eb1ab9ce4da766129e95710a4642501d37527 Oct 30 21:36:20.357187 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.357155 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.357265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.357231 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.357265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.357261 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:20.359290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:20.359269 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:36:21.053038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.053005 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" event={"ID":"7cd9dea5-a48f-404b-b888-809e9ac342b3","Type":"ContainerStarted","Data":"b2aec47519022a9eb592bc8565a2ed1b7ac27fe1cb8a1ff0d1e9a6017b474f5b"} Oct 30 21:36:21.053612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.053061 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" event={"ID":"7cd9dea5-a48f-404b-b888-809e9ac342b3","Type":"ContainerStarted","Data":"bb45dcad6a3b91cb28ebede5ee0eb1ab9ce4da766129e95710a4642501d37527"} Oct 30 21:36:21.054881 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.054857 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" event={"ID":"ade6ff81-281b-4e00-80d6-d6026fa257df","Type":"ContainerStarted","Data":"64e78a30ca860289ba808c7e9fa002d83b6a6a86c1b4e30c71d4691a6bec38e8"} Oct 30 21:36:21.055001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.054911 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" event={"ID":"ade6ff81-281b-4e00-80d6-d6026fa257df","Type":"ContainerStarted","Data":"1b25bfcb1dc6ff1ee8ec1467d2a8eb618d67e51d8acf4d9983bf1107c6130599"} Oct 30 21:36:21.172508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.172453 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podStartSLOduration=2.172439135 podStartE2EDuration="2.172439135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:21.169772247 +0000 UTC m=+1495.152472666" watchObservedRunningTime="2026-10-30 21:36:21.172439135 +0000 UTC m=+1495.155139600" Oct 30 21:36:21.295529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.294151 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podStartSLOduration=2.294135686 podStartE2EDuration="2.294135686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:21.293535203 +0000 UTC m=+1495.276235622" watchObservedRunningTime="2026-10-30 21:36:21.294135686 +0000 UTC m=+1495.276836103" Oct 30 21:36:21.872688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.872629 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:21.875479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.875439 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:21.875611 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.875496 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:21.930339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.921475 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:21.934539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.934492 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:21.934695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:21.934572 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:22.865392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:22.865218 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:22.865392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:22.865287 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:22.912368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:22.910694 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:22.912368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:22.910751 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:23.873344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:23.870414 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:23.873344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:23.870474 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:23.922343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:23.918837 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:23.922343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:23.918909 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:24.828339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.825132 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:36:24.872383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.871728 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:24.872383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.871794 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:24.891964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.876581 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.915848 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.915965 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916043 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrsw\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916151 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916211 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916291 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916339 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916441 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:24.929336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:24.916494 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.132428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.006208 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.132428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.006261 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:25.132428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.040767 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:36:25.254340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.220957 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.269112 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.269083 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.269481 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.269458 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.269690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.269674 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.270182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.270162 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.270420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.270392 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.270602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.270585 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrsw\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.270879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.270864 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.271022 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.271008 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.271484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.271464 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.271606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.227024 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.271919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.271903 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.272246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.271935 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.272370 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.272209 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.281091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.281008 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.284484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.284384 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.363500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.359502 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.363500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.360691 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrsw\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\") pod \"e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.490740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.490705 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:25.864396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.864225 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.864396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.864273 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:25.925342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.922798 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.925342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:25.922862 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:26.068793 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:36:26.068755 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006d8097_44b3_48d2_85f8_53ff8940234f.slice/crio-e7e18ac2e1d54dbcdf28d8ca96149482a60a3cecd654ecd33fb18c77197cfebd WatchSource:0}: Error finding container e7e18ac2e1d54dbcdf28d8ca96149482a60a3cecd654ecd33fb18c77197cfebd: Status 404 returned error can't find the container with id e7e18ac2e1d54dbcdf28d8ca96149482a60a3cecd654ecd33fb18c77197cfebd Oct 30 21:36:26.068926 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.068903 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:36:26.072121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.072079 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:26.072293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.072274 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:26.072446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.072428 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:26.283643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.281856 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" event={"ID":"006d8097-44b3-48d2-85f8-53ff8940234f","Type":"ContainerStarted","Data":"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a"} Oct 30 21:36:26.283643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.281922 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" event={"ID":"006d8097-44b3-48d2-85f8-53ff8940234f","Type":"ContainerStarted","Data":"e7e18ac2e1d54dbcdf28d8ca96149482a60a3cecd654ecd33fb18c77197cfebd"} Oct 30 21:36:26.459397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.447400 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podStartSLOduration=2.44737637 podStartE2EDuration="2.44737637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:26.420063679 +0000 UTC m=+1500.402764096" watchObservedRunningTime="2026-10-30 21:36:26.44737637 +0000 UTC m=+1500.430076788" Oct 30 21:36:26.491703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.491677 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:26.578699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.578672 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:36:26.579489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.579472 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:36:26.594770 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.594749 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:36:26.595971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.595956 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:36:26.642686 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.642643 4921 scope.go:122] "RemoveContainer" containerID="0a5e033805254fac998da65b91a3b53086f4464257a49d3ebcf3497e556b71a2" Oct 30 21:36:26.666740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.666713 4921 scope.go:122] "RemoveContainer" containerID="8fcc7c9677c0d4eae5696e68761df5928f994afc9a47da0ce3ce0caf9a50f3f3" Oct 30 21:36:26.864331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.864284 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:36:26.864525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.864361 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:36:26.932625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.920573 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:26.932625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:26.920633 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:27.491923 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:27.491891 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:27.492105 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:27.491947 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:27.915139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:27.914661 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:36:27.915139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:27.914717 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:36:28.466556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.466520 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:28.468269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.468239 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:28.469473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.469452 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:36:28.491495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.491372 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:28.491495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.491413 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:28.946672 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.946590 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:28.947556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.947487 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:28.949490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:28.949453 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:36:29.571491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:29.570527 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:29.571491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:29.570585 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:30.501276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:30.501241 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:30.501553 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:30.501518 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:31.282392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.279743 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:31.338935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.302398 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:31.440341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.431404 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:31.502903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.502870 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:31.531191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.531150 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:31.531849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.515558 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:31.633448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:31.619993 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:32.302216 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.297453 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:32.303145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.303117 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:32.325423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.325396 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:36:32.325663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.325647 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:36:32.336757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.336723 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 21:36:32.430423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.427649 4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:32.562396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.533859 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:32.562396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.533918 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:32.562396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.534012 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:36:32.562396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.534726 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:36:32.591341 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.584083 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:32.642385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.593017 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:36:32.654618 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.620336 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:36:32.669699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.669361 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:36:32.758231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:32.755869 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.310435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.310386 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.362924 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.337840 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.398217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.309602 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 21:36:33.398217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.364920 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:36:33.516122 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.516076 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.601465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.595894 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:33.601465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.595952 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:33.601465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.597791 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 21:36:33.601465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.599126 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 21:36:33.638810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.638773 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:36:33.639112 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.639040 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.675017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.674969 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 30 21:36:33.761553 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.761509 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:33.762063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:33.762042 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:36:34.020155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.020124 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:36:34.037335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.025430 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:36:34.054921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.054880 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:34.055931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.055909 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:36:34.405662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.405616 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:34.413610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.413582 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:36:34.430294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.417960 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:36:34.430294 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.421999 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:36:34.494063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.494019 4921 patch_prober.go:29] interesting pod/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:36:34.494373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.494344 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:36:34.505353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.505326 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:36:34.526133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:34.526101 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:36:35.907412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:35.907373 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:35.908851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:35.908823 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:35.909402 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:35.909381 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:36:37.550086 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:36:37.550049 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:36:55.356621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.356581 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 21:36:55.364695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.364670 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.417688 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.417662 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6bm8h\"" Oct 30 21:36:55.474942 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.474909 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 21:36:55.527798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.527769 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.527945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.527825 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.527945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.527882 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528013 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528058 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528126 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528190 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528176 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528279 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.528382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.528369 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630084 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630166 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630201 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630278 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630362 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630422 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630470 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630574 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630619 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630819 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630802 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.630972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.630703 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.631267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.631240 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.631399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.631358 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.632825 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.632802 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.632911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.632895 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.686166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.686146 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.686304 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.686283 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\") pod \"maas-default-gateway-openshift-default-6cc458d887-jq6qc\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:55.979518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:55.979491 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:56.418615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:56.418586 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 21:36:56.420472 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:36:56.420445 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76cebce_aa54_4518_ab9d_5e5530a1c3c2.slice/crio-44e571fae25d57bdf6dbca1f38b31c9575cb2b8db412429be119b9ae2c165e46 WatchSource:0}: Error finding container 44e571fae25d57bdf6dbca1f38b31c9575cb2b8db412429be119b9ae2c165e46: Status 404 returned error can't find the container with id 44e571fae25d57bdf6dbca1f38b31c9575cb2b8db412429be119b9ae2c165e46 Oct 30 21:36:56.423077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:56.423030 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:56.423198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:56.423119 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:56.423198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:56.423165 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:36:57.332668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.332630 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" event={"ID":"f76cebce-aa54-4518-ab9d-5e5530a1c3c2","Type":"ContainerStarted","Data":"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7"} Oct 30 21:36:57.332883 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.332685 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" event={"ID":"f76cebce-aa54-4518-ab9d-5e5530a1c3c2","Type":"ContainerStarted","Data":"44e571fae25d57bdf6dbca1f38b31c9575cb2b8db412429be119b9ae2c165e46"} Oct 30 21:36:57.499622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.499563 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podStartSLOduration=2.49954572 podStartE2EDuration="2.49954572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:57.498371171 +0000 UTC m=+1531.481071590" watchObservedRunningTime="2026-10-30 21:36:57.49954572 +0000 UTC m=+1531.482246139" Oct 30 21:36:57.979954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.979921 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:36:57.981106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.981082 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:36:57.981182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:57.981135 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:36:58.980156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:58.980114 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:36:58.980655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:58.980171 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:36:59.979795 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:59.979764 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:36:59.980031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:36:59.980003 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:37:00.980343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:00.980297 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:37:00.980805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:00.980388 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:37:01.979776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:01.979733 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:37:01.979974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:01.979796 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:37:02.980217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:02.980179 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:37:02.980616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:02.980248 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:37:04.849134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:04.849102 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:37:04.849664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:04.849495 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:37:04.850196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:04.850176 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 21:37:07.542838 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:37:07.542786 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:37:17.930094 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:17.930054 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv"] Oct 30 21:37:17.935091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:17.935069 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:17.985889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:17.985859 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv"] Oct 30 21:37:18.033636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.033606 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:18.033987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.033965 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:37:18.038421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.038383 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.038763 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.038725 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.039126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.039091 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlbt\" (UniqueName: \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.140364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.140282 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.140551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.140465 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.140631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.140600 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlbt\" (UniqueName: \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.141211 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.141182 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.143894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.143876 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.199486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.198641 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlbt\" (UniqueName: \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\") pod \"maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.253469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.253440 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:18.920346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:18.895934 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv"] Oct 30 21:37:19.690225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:19.690185 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" event={"ID":"6c723e6b-5270-4356-adfd-3f0d2d5cf52b","Type":"ContainerStarted","Data":"2242983b4330b02bafc4bf80c7945e5a7dfa6c91d4abd90690d21bd2930872d5"} Oct 30 21:37:19.690596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:19.690290 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" event={"ID":"6c723e6b-5270-4356-adfd-3f0d2d5cf52b","Type":"ContainerStarted","Data":"73a17127d1ba713d6635fe069934869c89c24d055c6f46c746d8a2b566685861"} Oct 30 21:37:19.827335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:19.826603 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" podStartSLOduration=2.82653276 podStartE2EDuration="2.82653276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:19.807691664 +0000 UTC m=+1553.790392088" watchObservedRunningTime="2026-10-30 21:37:19.82653276 +0000 UTC m=+1553.809233178" Oct 30 21:37:20.726340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:20.721743 4921 generic.go:356] "Generic (PLEG): container finished" podID="6c723e6b-5270-4356-adfd-3f0d2d5cf52b" containerID="2242983b4330b02bafc4bf80c7945e5a7dfa6c91d4abd90690d21bd2930872d5" exitCode=0 Oct 30 21:37:20.726340 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:20.721916 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" event={"ID":"6c723e6b-5270-4356-adfd-3f0d2d5cf52b","Type":"ContainerDied","Data":"2242983b4330b02bafc4bf80c7945e5a7dfa6c91d4abd90690d21bd2930872d5"} Oct 30 21:37:21.978774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:21.978743 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:22.032138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.032048 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\") pod \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " Oct 30 21:37:22.032299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.032166 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\") pod \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " Oct 30 21:37:22.032299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.032261 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\" (UniqueName: \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\") pod \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\" (UID: \"6c723e6b-5270-4356-adfd-3f0d2d5cf52b\") " Oct 30 21:37:22.032522 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.032481 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca" pod "6c723e6b-5270-4356-adfd-3f0d2d5cf52b" (UID: "6c723e6b-5270-4356-adfd-3f0d2d5cf52b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:22.033193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.033170 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:22.035167 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.035145 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt" pod "6c723e6b-5270-4356-adfd-3f0d2d5cf52b" (UID: "6c723e6b-5270-4356-adfd-3f0d2d5cf52b"). InnerVolumeSpecName "kube-api-access-mwlbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:22.036933 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.036907 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca" pod "6c723e6b-5270-4356-adfd-3f0d2d5cf52b" (UID: "6c723e6b-5270-4356-adfd-3f0d2d5cf52b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:22.135222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.135186 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:22.135222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.135220 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlbt\" (UniqueName: \"kubernetes.io/projected/6c723e6b-5270-4356-adfd-3f0d2d5cf52b-kube-api-access-mwlbt\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:22.748331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.748296 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" Oct 30 21:37:22.748490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.748365 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv" event={"ID":"6c723e6b-5270-4356-adfd-3f0d2d5cf52b","Type":"ContainerDied","Data":"73a17127d1ba713d6635fe069934869c89c24d055c6f46c746d8a2b566685861"} Oct 30 21:37:22.748490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:22.748423 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a17127d1ba713d6635fe069934869c89c24d055c6f46c746d8a2b566685861" Oct 30 21:37:23.474477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:23.474437 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:37:23.474982 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:23.474507 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:37:26.784911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.784879 4921 generic.go:356] "Generic (PLEG): container finished" podID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerID="dba0201b200ae739ccaa7e8c8dc53a3ba3a5ef8aaf70d6189d1b643743499cc5" exitCode=0 Oct 30 21:37:26.785356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.784953 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerDied","Data":"dba0201b200ae739ccaa7e8c8dc53a3ba3a5ef8aaf70d6189d1b643743499cc5"} Oct 30 21:37:26.785428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.785379 4921 scope.go:122] "RemoveContainer" containerID="dba0201b200ae739ccaa7e8c8dc53a3ba3a5ef8aaf70d6189d1b643743499cc5" Oct 30 21:37:26.786916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.786889 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:26.786984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.786955 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:26.787018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:26.786985 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:37:27.797099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:27.797067 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerStarted","Data":"824ff46b237077b410893d292ce23e81ddb06f105cbdc3e34ae4383617e22f01"} Oct 30 21:37:28.369135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:28.369100 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:37:28.369348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:28.369295 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:37:28.369425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:28.369353 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:37:29.369172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:29.369137 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:37:29.369560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:29.369192 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:37:30.129828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:30.129790 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:37:30.369346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:30.369292 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:37:30.369735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:30.369355 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:37:31.369163 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:31.369130 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:37:31.369360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:31.369187 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:37:32.368823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:32.368786 4921 patch_prober.go:29] interesting pod/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 21:37:32.369443 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:32.369405 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 21:37:33.455265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:33.455225 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:37:33.456055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:33.456031 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:37:33.459971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:33.459950 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:37:35.131228 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.131193 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" podUID="426afed3-ad13-4c86-a441-ea9babcbcc00" containerName="payload-processing" containerID="cri-o://87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c" gracePeriod=25 Oct 30 21:37:35.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.416376 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:37:35.524546 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.524509 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\" (UniqueName: \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\") pod \"426afed3-ad13-4c86-a441-ea9babcbcc00\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " Oct 30 21:37:35.525503 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.525476 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\") pod \"426afed3-ad13-4c86-a441-ea9babcbcc00\" (UID: \"426afed3-ad13-4c86-a441-ea9babcbcc00\") " Oct 30 21:37:35.526161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.525991 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume" pod "426afed3-ad13-4c86-a441-ea9babcbcc00" (UID: "426afed3-ad13-4c86-a441-ea9babcbcc00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:35.527018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.526989 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426afed3-ad13-4c86-a441-ea9babcbcc00-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:35.560607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.560558 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft" pod "426afed3-ad13-4c86-a441-ea9babcbcc00" (UID: "426afed3-ad13-4c86-a441-ea9babcbcc00"). InnerVolumeSpecName "kube-api-access-ctqft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:35.642927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.642837 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqft\" (UniqueName: \"kubernetes.io/projected/426afed3-ad13-4c86-a441-ea9babcbcc00-kube-api-access-ctqft\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:35.925969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.925881 4921 generic.go:356] "Generic (PLEG): container finished" podID="426afed3-ad13-4c86-a441-ea9babcbcc00" containerID="87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c" exitCode=0 Oct 30 21:37:35.925969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.925923 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" Oct 30 21:37:35.926145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.925967 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" event={"ID":"426afed3-ad13-4c86-a441-ea9babcbcc00","Type":"ContainerDied","Data":"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c"} Oct 30 21:37:35.926145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.926036 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5" event={"ID":"426afed3-ad13-4c86-a441-ea9babcbcc00","Type":"ContainerDied","Data":"bb3ed60a2bf9d1770bbe2e4bdeb437c7217068e8cec2d59494c5124f127fcd00"} Oct 30 21:37:35.926145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.926080 4921 scope.go:122] "RemoveContainer" containerID="87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c" Oct 30 21:37:35.938210 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.938187 4921 scope.go:122] "RemoveContainer" containerID="87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c" Oct 30 21:37:35.938541 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:37:35.938516 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c\": container with ID starting with 87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c not found: ID does not exist" containerID="87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c" Oct 30 21:37:35.938655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:35.938542 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c"} err="failed to get container status \"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c\": rpc error: code = NotFound desc = could not find container \"87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c\": container with ID starting with 87077b39643d6a52b6fd8724c64747558476d4ac54a504eaec01556569b29f2c not found: ID does not exist" Oct 30 21:37:36.091104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:36.091080 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:37:36.095815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:36.095790 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b25d536b-6bd4565df7-swpt5"] Oct 30 21:37:36.170029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:36.170001 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426afed3-ad13-4c86-a441-ea9babcbcc00" path="/var/lib/kubelet/pods/426afed3-ad13-4c86-a441-ea9babcbcc00/volumes" Oct 30 21:37:37.550560 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:37:37.550474 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:37:38.115784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.115754 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp"] Oct 30 21:37:38.116469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116452 4921 cpu_manager.go:414] "containerMap: removing container" podUID="426afed3-ad13-4c86-a441-ea9babcbcc00" containerName="payload-processing" Oct 30 21:37:38.116529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116470 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426afed3-ad13-4c86-a441-ea9babcbcc00" containerName="payload-processing" Oct 30 21:37:38.116529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116500 4921 cpu_manager.go:414] "containerMap: removing container" podUID="6c723e6b-5270-4356-adfd-3f0d2d5cf52b" containerName="revoke-keys" Oct 30 21:37:38.116529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116505 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c723e6b-5270-4356-adfd-3f0d2d5cf52b" containerName="revoke-keys" Oct 30 21:37:38.116637 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116627 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="426afed3-ad13-4c86-a441-ea9babcbcc00" containerName="payload-processing" Oct 30 21:37:38.116671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.116643 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c723e6b-5270-4356-adfd-3f0d2d5cf52b" containerName="revoke-keys" Oct 30 21:37:38.119678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.119664 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.177145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.177113 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp"] Oct 30 21:37:38.184500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.184473 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.184645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.184506 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788nl\" (UniqueName: \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.184645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.184615 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.220561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.220531 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:38.220710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.220690 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:37:38.285975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.285944 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.285975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.285982 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788nl\" (UniqueName: \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.286707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.286107 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.287651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.287483 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.290707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.290683 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.341980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.341956 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788nl\" (UniqueName: \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\") pod \"maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.432877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.432806 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:38.820160 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:37:38.820129 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad845405_275e_46f1_85b8_a67c4ada422e.slice/crio-31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e WatchSource:0}: Error finding container 31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e: Status 404 returned error can't find the container with id 31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e Oct 30 21:37:38.821307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.821280 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp"] Oct 30 21:37:38.953746 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:38.953717 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" event={"ID":"ad845405-275e-46f1-85b8-a67c4ada422e","Type":"ContainerStarted","Data":"31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e"} Oct 30 21:37:39.240949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.240924 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2"] Oct 30 21:37:39.244405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.244390 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.299130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.299100 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.299267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.299170 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7g2w\" (UniqueName: \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.299356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.299280 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.304357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.304296 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2"] Oct 30 21:37:39.401038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.400965 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.401216 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.401199 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.401286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.401263 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7g2w\" (UniqueName: \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.401793 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.401773 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.404093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.404070 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.458502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.458477 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7g2w\" (UniqueName: \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\") pod \"maas-api-revoke-sub-d6701073-jtxd2\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.557614 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.557588 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:39.946052 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.946028 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2"] Oct 30 21:37:39.947034 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:37:39.947011 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500c841f_479d_460c_8c3b_4417dcf00321.slice/crio-1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438 WatchSource:0}: Error finding container 1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438: Status 404 returned error can't find the container with id 1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438 Oct 30 21:37:39.962715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.962692 4921 generic.go:356] "Generic (PLEG): container finished" podID="ad845405-275e-46f1-85b8-a67c4ada422e" containerID="6f56ed499d5d5467287ddecf868d23ecd66d1692c2f43d3976747a74abc189ee" exitCode=0 Oct 30 21:37:39.962797 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.962774 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" event={"ID":"ad845405-275e-46f1-85b8-a67c4ada422e","Type":"ContainerDied","Data":"6f56ed499d5d5467287ddecf868d23ecd66d1692c2f43d3976747a74abc189ee"} Oct 30 21:37:39.964300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:39.964280 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" event={"ID":"500c841f-479d-460c-8c3b-4417dcf00321","Type":"ContainerStarted","Data":"1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438"} Oct 30 21:37:40.974372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:40.974342 4921 generic.go:356] "Generic (PLEG): container finished" podID="500c841f-479d-460c-8c3b-4417dcf00321" containerID="8a6c1ed45a3d574fb6086d11ae557b8c4c16584f358c35f35040e8a18c969563" exitCode=0 Oct 30 21:37:40.974760 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:40.974410 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" event={"ID":"500c841f-479d-460c-8c3b-4417dcf00321","Type":"ContainerDied","Data":"8a6c1ed45a3d574fb6086d11ae557b8c4c16584f358c35f35040e8a18c969563"} Oct 30 21:37:41.160420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.160392 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:41.226302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.226271 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\" (UniqueName: \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\") pod \"ad845405-275e-46f1-85b8-a67c4ada422e\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " Oct 30 21:37:41.226430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.226311 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\") pod \"ad845405-275e-46f1-85b8-a67c4ada422e\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " Oct 30 21:37:41.226430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.226413 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\") pod \"ad845405-275e-46f1-85b8-a67c4ada422e\" (UID: \"ad845405-275e-46f1-85b8-a67c4ada422e\") " Oct 30 21:37:41.226735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.226708 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca" pod "ad845405-275e-46f1-85b8-a67c4ada422e" (UID: "ad845405-275e-46f1-85b8-a67c4ada422e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:41.227118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.227092 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:41.228447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.228419 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca" pod "ad845405-275e-46f1-85b8-a67c4ada422e" (UID: "ad845405-275e-46f1-85b8-a67c4ada422e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:41.228524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.228483 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl" pod "ad845405-275e-46f1-85b8-a67c4ada422e" (UID: "ad845405-275e-46f1-85b8-a67c4ada422e"). InnerVolumeSpecName "kube-api-access-788nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:41.328441 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.328377 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788nl\" (UniqueName: \"kubernetes.io/projected/ad845405-275e-46f1-85b8-a67c4ada422e-kube-api-access-788nl\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:41.328441 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.328401 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad845405-275e-46f1-85b8-a67c4ada422e-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:41.410521 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.410491 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn"] Oct 30 21:37:41.411243 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.411228 4921 cpu_manager.go:414] "containerMap: removing container" podUID="ad845405-275e-46f1-85b8-a67c4ada422e" containerName="revoke-keys" Oct 30 21:37:41.411287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.411245 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad845405-275e-46f1-85b8-a67c4ada422e" containerName="revoke-keys" Oct 30 21:37:41.411413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.411402 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad845405-275e-46f1-85b8-a67c4ada422e" containerName="revoke-keys" Oct 30 21:37:41.413895 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.413874 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.472752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.472725 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn"] Oct 30 21:37:41.530856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.530810 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.531051 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.530968 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.531121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.531076 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvntj\" (UniqueName: \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.633161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.633080 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.633307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.633173 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvntj\" (UniqueName: \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.633448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.633431 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.634042 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.634013 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.635500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.635479 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.688980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.688953 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvntj\" (UniqueName: \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\") pod \"maas-api-revoke-keys-e2e-derive-6f787654-c7lmn\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.726877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.726854 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:41.985254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.985227 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" Oct 30 21:37:41.985701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.985223 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp" event={"ID":"ad845405-275e-46f1-85b8-a67c4ada422e","Type":"ContainerDied","Data":"31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e"} Oct 30 21:37:41.985701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:41.985390 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d39fe68cdc88e9e2eeaa4496c7a169eb375460bdef26a8d0acc14f83bb0f8e" Oct 30 21:37:42.113612 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:37:42.113569 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6472a0_b26e_4261_9851_ecfc606c780e.slice/crio-816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d WatchSource:0}: Error finding container 816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d: Status 404 returned error can't find the container with id 816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d Oct 30 21:37:42.115152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.115124 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn"] Oct 30 21:37:42.132705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.132689 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:42.242658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.242608 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\" (UniqueName: \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\") pod \"500c841f-479d-460c-8c3b-4417dcf00321\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " Oct 30 21:37:42.242745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.242700 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\") pod \"500c841f-479d-460c-8c3b-4417dcf00321\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " Oct 30 21:37:42.242887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.242869 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\") pod \"500c841f-479d-460c-8c3b-4417dcf00321\" (UID: \"500c841f-479d-460c-8c3b-4417dcf00321\") " Oct 30 21:37:42.243215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.243194 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca" pod "500c841f-479d-460c-8c3b-4417dcf00321" (UID: "500c841f-479d-460c-8c3b-4417dcf00321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:42.243557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.243532 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:42.244755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.244734 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca" pod "500c841f-479d-460c-8c3b-4417dcf00321" (UID: "500c841f-479d-460c-8c3b-4417dcf00321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:42.244840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.244812 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w" pod "500c841f-479d-460c-8c3b-4417dcf00321" (UID: "500c841f-479d-460c-8c3b-4417dcf00321"). InnerVolumeSpecName "kube-api-access-j7g2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:42.345034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.345004 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7g2w\" (UniqueName: \"kubernetes.io/projected/500c841f-479d-460c-8c3b-4417dcf00321-kube-api-access-j7g2w\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:42.345034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.345033 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500c841f-479d-460c-8c3b-4417dcf00321-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:42.994848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.994809 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" event={"ID":"2d6472a0-b26e-4261-9851-ecfc606c780e","Type":"ContainerStarted","Data":"d40c195f43fa03b6b897c9d7b76a49353f228929c03805a1e67a534429c8bffe"} Oct 30 21:37:42.995337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.994875 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" event={"ID":"2d6472a0-b26e-4261-9851-ecfc606c780e","Type":"ContainerStarted","Data":"816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d"} Oct 30 21:37:42.996348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.996305 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" event={"ID":"500c841f-479d-460c-8c3b-4417dcf00321","Type":"ContainerDied","Data":"1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438"} Oct 30 21:37:42.996447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.996349 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2" Oct 30 21:37:42.996447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:42.996364 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f887d08b2fc2db4f0a0930f2a841ed8b79da10c0fa160948091c69dbd63b438" Oct 30 21:37:43.112505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:43.112454 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" podStartSLOduration=2.112439223 podStartE2EDuration="2.112439223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:43.108177514 +0000 UTC m=+1577.090877932" watchObservedRunningTime="2026-10-30 21:37:43.112439223 +0000 UTC m=+1577.095139640" Oct 30 21:37:44.007231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:44.007195 4921 generic.go:356] "Generic (PLEG): container finished" podID="2d6472a0-b26e-4261-9851-ecfc606c780e" containerID="d40c195f43fa03b6b897c9d7b76a49353f228929c03805a1e67a534429c8bffe" exitCode=0 Oct 30 21:37:44.007669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:44.007285 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" event={"ID":"2d6472a0-b26e-4261-9851-ecfc606c780e","Type":"ContainerDied","Data":"d40c195f43fa03b6b897c9d7b76a49353f228929c03805a1e67a534429c8bffe"} Oct 30 21:37:45.196422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.196393 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:45.312582 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.312472 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\") pod \"2d6472a0-b26e-4261-9851-ecfc606c780e\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " Oct 30 21:37:45.312879 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.312849 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\") pod \"2d6472a0-b26e-4261-9851-ecfc606c780e\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " Oct 30 21:37:45.313011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.312894 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\" (UniqueName: \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\") pod \"2d6472a0-b26e-4261-9851-ecfc606c780e\" (UID: \"2d6472a0-b26e-4261-9851-ecfc606c780e\") " Oct 30 21:37:45.314576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.314552 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca" pod "2d6472a0-b26e-4261-9851-ecfc606c780e" (UID: "2d6472a0-b26e-4261-9851-ecfc606c780e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:45.316894 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.316863 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca" pod "2d6472a0-b26e-4261-9851-ecfc606c780e" (UID: "2d6472a0-b26e-4261-9851-ecfc606c780e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:45.317141 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.317115 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj" pod "2d6472a0-b26e-4261-9851-ecfc606c780e" (UID: "2d6472a0-b26e-4261-9851-ecfc606c780e"). InnerVolumeSpecName "kube-api-access-dvntj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:45.415259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.415222 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:45.415259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.415258 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6472a0-b26e-4261-9851-ecfc606c780e-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:45.415473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:45.415271 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvntj\" (UniqueName: \"kubernetes.io/projected/2d6472a0-b26e-4261-9851-ecfc606c780e-kube-api-access-dvntj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:46.024639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:46.024601 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" Oct 30 21:37:46.024813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:46.024599 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn" event={"ID":"2d6472a0-b26e-4261-9851-ecfc606c780e","Type":"ContainerDied","Data":"816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d"} Oct 30 21:37:46.024813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:46.024727 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="816bfaf66a10bdcc7b161a17bb5b1d7571f8bd29f5a72def6eef1f9a2c4f1a7d" Oct 30 21:37:49.706564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:49.706525 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:37:54.707256 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:54.707211 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" podUID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" containerName="payload-pre-processing" containerID="cri-o://5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b" gracePeriod=25 Oct 30 21:37:54.995289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:54.995258 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:37:55.148207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.148174 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\" (UniqueName: \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\") pod \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " Oct 30 21:37:55.148423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.148409 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\") pod \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\" (UID: \"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8\") " Oct 30 21:37:55.148777 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.148746 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume" pod "77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" (UID: "77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:55.149099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.149080 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:55.150277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.150254 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg" pod "77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" (UID: "77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8"). InnerVolumeSpecName "kube-api-access-v22bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:55.241552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.241487 4921 generic.go:356] "Generic (PLEG): container finished" podID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" containerID="5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b" exitCode=0 Oct 30 21:37:55.241552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.241534 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" Oct 30 21:37:55.241710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.241576 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" event={"ID":"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8","Type":"ContainerDied","Data":"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b"} Oct 30 21:37:55.241710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.241631 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44" event={"ID":"77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8","Type":"ContainerDied","Data":"f31d081024449f8f2010a09ae6a6e1a197c7e14a09276a2f52daa4aae6b3373e"} Oct 30 21:37:55.241710 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.241660 4921 scope.go:122] "RemoveContainer" containerID="5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b" Oct 30 21:37:55.250253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.250228 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v22bg\" (UniqueName: \"kubernetes.io/projected/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8-kube-api-access-v22bg\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:55.256466 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.256412 4921 scope.go:122] "RemoveContainer" containerID="5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b" Oct 30 21:37:55.257618 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:37:55.256788 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b\": container with ID starting with 5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b not found: ID does not exist" containerID="5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b" Oct 30 21:37:55.257618 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.256824 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b"} err="failed to get container status \"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b\": rpc error: code = NotFound desc = could not find container \"5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b\": container with ID starting with 5bffe3eb0e8e0eddaeafc12258adb3385b6336f60105ed4c87a3a3b23ca0992b not found: ID does not exist" Oct 30 21:37:55.406681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.406649 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:37:55.411396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:55.411371 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d5266a5-5f46b4cfc9-hhb44"] Oct 30 21:37:56.181971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:37:56.181939 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" path="/var/lib/kubelet/pods/77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8/volumes" Oct 30 21:38:07.540657 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:38:07.540624 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:38:11.690608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:11.690569 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv"] Oct 30 21:38:11.740530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:11.740500 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b25d536b-jb9dv"] Oct 30 21:38:12.171908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:12.171874 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c723e6b-5270-4356-adfd-3f0d2d5cf52b" path="/var/lib/kubelet/pods/6c723e6b-5270-4356-adfd-3f0d2d5cf52b/volumes" Oct 30 21:38:17.911187 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:17.911148 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp"] Oct 30 21:38:17.941213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:17.941182 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d5266a5-zr6fp"] Oct 30 21:38:18.171082 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:18.171007 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad845405-275e-46f1-85b8-a67c4ada422e" path="/var/lib/kubelet/pods/ad845405-275e-46f1-85b8-a67c4ada422e/volumes" Oct 30 21:38:24.307804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:24.307773 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn"] Oct 30 21:38:24.360424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:24.360384 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f787654-c7lmn"] Oct 30 21:38:24.366325 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:24.366286 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:38:24.366569 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:24.366544 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" containerID="cri-o://824ff46b237077b410893d292ce23e81ddb06f105cbdc3e34ae4383617e22f01" gracePeriod=30 Oct 30 21:38:26.131077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131042 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:38:26.131757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131740 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2d6472a0-b26e-4261-9851-ecfc606c780e" containerName="revoke-keys" Oct 30 21:38:26.131757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131756 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6472a0-b26e-4261-9851-ecfc606c780e" containerName="revoke-keys" Oct 30 21:38:26.131898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131779 4921 cpu_manager.go:414] "containerMap: removing container" podUID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" containerName="payload-pre-processing" Oct 30 21:38:26.131898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131784 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" containerName="payload-pre-processing" Oct 30 21:38:26.131898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131813 4921 cpu_manager.go:414] "containerMap: removing container" podUID="500c841f-479d-460c-8c3b-4417dcf00321" containerName="revoke-keys" Oct 30 21:38:26.131898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131818 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500c841f-479d-460c-8c3b-4417dcf00321" containerName="revoke-keys" Oct 30 21:38:26.132090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131907 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c8c8f2-1e0e-4e6d-ab82-490b77dfcad8" containerName="payload-pre-processing" Oct 30 21:38:26.132090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131921 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="500c841f-479d-460c-8c3b-4417dcf00321" containerName="revoke-keys" Oct 30 21:38:26.132090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.131930 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6472a0-b26e-4261-9851-ecfc606c780e" containerName="revoke-keys" Oct 30 21:38:26.135597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.135577 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.172462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.172439 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6472a0-b26e-4261-9851-ecfc606c780e" path="/var/lib/kubelet/pods/2d6472a0-b26e-4261-9851-ecfc606c780e/volumes" Oct 30 21:38:26.179905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.179885 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:38:26.184029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.184014 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.232347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.232322 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249306 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249383 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249421 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249481 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249506 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65k4g\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249573 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249618 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249694 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249729 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249776 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249812 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr54l\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249888 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249909 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.249938 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.250014 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.250077 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.250143 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.250175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.251049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.251033 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:38:26.351840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351804 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351851 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr54l\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351905 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351921 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351939 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.351992 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352052 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352097 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352127 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352220 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352244 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352271 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352309 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352425 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352438 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352453 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65k4g\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352528 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352531 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352626 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352699 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352721 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352729 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.352814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352762 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.353545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352962 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.353545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.352982 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.353545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.353393 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.353706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.353540 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.353706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.353577 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.354772 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.354747 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.354969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.354953 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.355154 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.355140 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.355508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.355488 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.409367 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.409272 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.409547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.409525 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.410116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.410094 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr54l\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\") pod \"maas-default-gateway-openshift-default-6cc458d887-8l67h\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.410669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.410645 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65k4g\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\") pod \"maas-default-gateway-openshift-default-6cc458d887-2vlfj\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.449251 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.449226 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:26.501633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.501604 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:26.886536 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:38:26.886502 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ca85d_68a4_46e4_9198_2c803b86742d.slice/crio-ef49d40de29993bc4fdef8942f95c800c9f14e0d522f888dec577e3e18fbf386 WatchSource:0}: Error finding container ef49d40de29993bc4fdef8942f95c800c9f14e0d522f888dec577e3e18fbf386: Status 404 returned error can't find the container with id ef49d40de29993bc4fdef8942f95c800c9f14e0d522f888dec577e3e18fbf386 Oct 30 21:38:26.887794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.887770 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:38:26.888801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.888768 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.888870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.888830 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.888870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.888865 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.936242 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:38:26.936215 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb774724_92b3_48f7_8d62_fc5c321ad16c.slice/crio-6c561d7902b7ed95ac5507fb7973dc1e0c2591cfb654a2eb4ea726619c51f709 WatchSource:0}: Error finding container 6c561d7902b7ed95ac5507fb7973dc1e0c2591cfb654a2eb4ea726619c51f709: Status 404 returned error can't find the container with id 6c561d7902b7ed95ac5507fb7973dc1e0c2591cfb654a2eb4ea726619c51f709 Oct 30 21:38:26.939070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.939029 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.939183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.939115 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.939246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.939183 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:38:26.942388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:26.942328 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:38:27.651296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.650951 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" event={"ID":"9f4ca85d-68a4-46e4-9198-2c803b86742d","Type":"ContainerStarted","Data":"f422bc0711937369f7e8cc854a4b12f37209ebe144d57775916d775a0c5d6ce6"} Oct 30 21:38:27.651296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.651016 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" event={"ID":"9f4ca85d-68a4-46e4-9198-2c803b86742d","Type":"ContainerStarted","Data":"ef49d40de29993bc4fdef8942f95c800c9f14e0d522f888dec577e3e18fbf386"} Oct 30 21:38:27.655561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.655412 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" event={"ID":"eb774724-92b3-48f7-8d62-fc5c321ad16c","Type":"ContainerStarted","Data":"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b"} Oct 30 21:38:27.655561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.655472 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" event={"ID":"eb774724-92b3-48f7-8d62-fc5c321ad16c","Type":"ContainerStarted","Data":"6c561d7902b7ed95ac5507fb7973dc1e0c2591cfb654a2eb4ea726619c51f709"} Oct 30 21:38:27.785216 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.785046 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podStartSLOduration=1.78502984 podStartE2EDuration="1.78502984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:27.778020823 +0000 UTC m=+1621.760721242" watchObservedRunningTime="2026-10-30 21:38:27.78502984 +0000 UTC m=+1621.767730258" Oct 30 21:38:27.937407 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:27.933171 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podStartSLOduration=1.933150997 podStartE2EDuration="1.933150997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:27.930672155 +0000 UTC m=+1621.913372574" watchObservedRunningTime="2026-10-30 21:38:27.933150997 +0000 UTC m=+1621.915851416" Oct 30 21:38:28.450586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.450433 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:28.451889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.451860 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:28.452078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.452052 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:28.520368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.519913 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:28.520368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.520049 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:28.520368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:28.520088 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:29.451134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.450676 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:29.451134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.450735 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:29.534338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.523699 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:29.534338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.523753 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:29.694175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.694144 4921 generic.go:356] "Generic (PLEG): container finished" podID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerID="824ff46b237077b410893d292ce23e81ddb06f105cbdc3e34ae4383617e22f01" exitCode=0 Oct 30 21:38:29.694721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.694386 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerDied","Data":"824ff46b237077b410893d292ce23e81ddb06f105cbdc3e34ae4383617e22f01"} Oct 30 21:38:29.694836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.694765 4921 scope.go:122] "RemoveContainer" containerID="dba0201b200ae739ccaa7e8c8dc53a3ba3a5ef8aaf70d6189d1b643743499cc5" Oct 30 21:38:29.883994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:29.883969 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082470 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082558 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082588 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082689 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082761 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082857 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.082979 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.083049 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.083335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.083124 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\") pod \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\" (UID: \"4a769d7c-7beb-481b-af2f-10c8d5f2ff80\") " Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.086662 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.086845 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.087055 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.087352 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.087412 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:30.087455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.087455 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:30.090783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.090756 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "kube-api-access-djpf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:30.091106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.091084 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:30.091784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.091742 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token" pod "4a769d7c-7beb-481b-af2f-10c8d5f2ff80" (UID: "4a769d7c-7beb-481b-af2f-10c8d5f2ff80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186592 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186624 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186640 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186656 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186670 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186684 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186698 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186713 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.187568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.186727 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpf2\" (UniqueName: \"kubernetes.io/projected/4a769d7c-7beb-481b-af2f-10c8d5f2ff80-kube-api-access-djpf2\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:30.497376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.488423 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:30.497376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.488487 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:30.529348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.520770 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:30.529348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.520829 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:30.762984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.762904 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" event={"ID":"4a769d7c-7beb-481b-af2f-10c8d5f2ff80","Type":"ContainerDied","Data":"4ada47b73934c6e6a4242d93f41545609689541b25b8e03c6662024541fa2061"} Oct 30 21:38:30.762984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.762935 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb" Oct 30 21:38:30.762984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.762966 4921 scope.go:122] "RemoveContainer" containerID="824ff46b237077b410893d292ce23e81ddb06f105cbdc3e34ae4383617e22f01" Oct 30 21:38:30.959353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.957801 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:38:30.967098 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:30.967070 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6f787654-openshift-default-6c6b7dfb9c-z2mdb"] Oct 30 21:38:31.451592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:31.451559 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:31.451882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:31.451851 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:31.516361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:31.515448 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:31.516361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:31.515506 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:32.176024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:32.175993 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" path="/var/lib/kubelet/pods/4a769d7c-7beb-481b-af2f-10c8d5f2ff80/volumes" Oct 30 21:38:32.453886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:32.452550 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:32.453886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:32.452627 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:32.502713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:32.502678 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:32.503139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:32.503105 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:33.449679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:33.449639 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:33.450155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:33.449713 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:33.502584 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:33.502548 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:33.502753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:33.502606 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:34.450054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:34.450016 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:38:34.450452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:34.450089 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:38:34.502111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:34.502080 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:38:34.502278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:34.502133 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 21:38:35.863164 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:35.863134 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:35.863798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:35.863772 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:35.864918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:35.864879 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:38:36.036706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:36.036676 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:36.037297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:36.037244 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:36.040853 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:38:36.040833 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:38:37.541149 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:38:37.541109 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:39:07.541035 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:39:07.540955 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:39:37.541276 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:39:37.541232 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:40:05.071801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:05.071764 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc"] Oct 30 21:40:05.105594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:05.105563 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981f88fa-cj8sc"] Oct 30 21:40:06.173798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:06.173767 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9469a73a-3eb3-4011-8b25-3da08abc2479" path="/var/lib/kubelet/pods/9469a73a-3eb3-4011-8b25-3da08abc2479/volumes" Oct 30 21:40:07.540705 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:40:07.540671 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:40:11.054335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:11.054291 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt"] Oct 30 21:40:11.103508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:11.103477 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c88e6742-c4kbt"] Oct 30 21:40:12.171225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:12.171194 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be3b305-61ee-43f4-82e9-710b2e07628c" path="/var/lib/kubelet/pods/6be3b305-61ee-43f4-82e9-710b2e07628c/volumes" Oct 30 21:40:20.098339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:20.097941 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974"] Oct 30 21:40:20.142338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:20.141403 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28b059e-gv974"] Oct 30 21:40:20.193697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:20.193660 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08b4a33-3e00-4034-8f2f-e2e182fe3f76" path="/var/lib/kubelet/pods/b08b4a33-3e00-4034-8f2f-e2e182fe3f76/volumes" Oct 30 21:40:20.893493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:20.893438 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:20.893701 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:20.893551 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.548358 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:40:37.548228 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:40:38.985396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:38.981247 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.073966 4921 cpu_manager.go:414] "containerMap: removing container" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.073996 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.074012 4921 cpu_manager.go:414] "containerMap: removing container" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.074020 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.074248 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.077093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.074259 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a769d7c-7beb-481b-af2f-10c8d5f2ff80" containerName="istio-proxy" Oct 30 21:40:39.165336 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.104261 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.267527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.267356 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.267882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.267580 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.267882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.267758 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.270170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.267992 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.270170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.268097 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.270170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.268152 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.270170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.268204 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.270170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.268264 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.271419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.271389 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8mz\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.272063 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.272039 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:40:39.304779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.304758 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-815dab-openshift-default-dockercfg-g6zvb\"" Oct 30 21:40:39.373222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373189 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373337 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373410 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373565 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373469 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373565 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373513 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373565 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373559 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373593 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373667 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8mz\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373724 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.373912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.373888 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.374144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.374121 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.374221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.374178 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.374221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.374183 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.374495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.374474 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.375876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.375851 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.376129 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.376114 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.454776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.454744 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.455778 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.455748 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8mz\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\") pod \"e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:39.580227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:39.580152 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:40.035438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.035392 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:40:40.072636 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:40:40.072599 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7bb186_b1d7_461f_a323_e3b945404994.slice/crio-ed080986706fda104265c36388c4a9af455131f45a6abf24ea5d3fa27adda137 WatchSource:0}: Error finding container ed080986706fda104265c36388c4a9af455131f45a6abf24ea5d3fa27adda137: Status 404 returned error can't find the container with id ed080986706fda104265c36388c4a9af455131f45a6abf24ea5d3fa27adda137 Oct 30 21:40:40.075404 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.075368 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:40.075485 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.075442 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:40.075485 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.075470 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:40.327303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.327214 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerStarted","Data":"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7"} Oct 30 21:40:40.327303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.327275 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerStarted","Data":"ed080986706fda104265c36388c4a9af455131f45a6abf24ea5d3fa27adda137"} Oct 30 21:40:40.453959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.453908 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podStartSLOduration=2.45389363 podStartE2EDuration="2.45389363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:40.452032854 +0000 UTC m=+1754.434733289" watchObservedRunningTime="2026-10-30 21:40:40.45389363 +0000 UTC m=+1754.436594109" Oct 30 21:40:40.584130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:40.583990 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:41.586911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:41.586875 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:42.345393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:42.345358 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:40:42.346331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:40:42.346298 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:41:07.542779 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:07.542740 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:41:12.329841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.329804 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:41:12.330261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.330006 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" containerID="cri-o://8eb80001e2b12bfdd43b840c07db76b806c54242752d4c649bccc8a7e948f5f1" gracePeriod=30 Oct 30 21:41:12.381906 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.381873 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:41:12.382139 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.382116 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" containerID="cri-o://f3ca8a9c6502568cbf62655ca97c46e9fea32707cceca28db9636aa3dc87baff" gracePeriod=30 Oct 30 21:41:12.389423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.389395 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:41:12.389634 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.389608 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" containerID="cri-o://80c9f3c7b7ffac08ae619873c3c74619adf56a1557bd870774a5d3b507ca5622" gracePeriod=30 Oct 30 21:41:12.429149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.429119 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:41:12.433729 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.433703 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:41:12.433952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.433928 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" containerID="cri-o://94e02d49cc0a06a731a452e4f8a222d58e058ff7d74ec57f61c246c0bcc9b64e" gracePeriod=30 Oct 30 21:41:12.439132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.439107 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:41:12.439361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.439306 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" containerID="cri-o://20e4e4320fbde60c71abd348c1913366c5c8d7f9b4fa31afb0ea0384c12e38d6" gracePeriod=30 Oct 30 21:41:12.480305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.480270 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:41:12.485107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.485077 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:41:12.485461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.485332 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" containerID="cri-o://0e83ec614700c345a315b6b6f94e080e358027a6b2efe5fcc2ef8b3034ac3fb5" gracePeriod=30 Oct 30 21:41:12.490202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.490011 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:41:12.490384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.490303 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" containerID="cri-o://2ec0902066a4095e31c473dccba37130cb05dadee3ba375a7a23ac1a7f41d923" gracePeriod=30 Oct 30 21:41:12.496122 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.495995 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:41:12.496265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.496185 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" containerID="cri-o://faa2111eecc9e4c3ef9a29c0d465d997a120d23eb52c928407c30c3c384263f0" gracePeriod=30 Oct 30 21:41:12.500709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.500684 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:41:12.500905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.500882 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" containerID="cri-o://beab3a31d27a8587f2b8fbe0e49f695bc4b1942fed0edd469616fa56c2503858" gracePeriod=30 Oct 30 21:41:12.655275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.655180 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" containerID="cri-o://1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2" gracePeriod=30 Oct 30 21:41:12.655275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:12.655213 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" containerID="cri-o://bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d" gracePeriod=30 Oct 30 21:41:16.090918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.090887 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:41:16.091335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.090935 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:41:16.286261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.286221 4921 patch_prober.go:29] interesting pod/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:41:16.286448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.286292 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:41:16.373176 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.373096 4921 patch_prober.go:29] interesting pod/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:41:16.373352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.373167 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:41:16.383679 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.383649 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:41:16.388547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.388514 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.447376 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.447302 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-be8b73-serving-cert\"" Oct 30 21:41:16.451810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.451783 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:41:16.529141 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.529107 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.529310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.529267 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.529310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.529343 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.529660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.529629 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fxv\" (UniqueName: \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.631647 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.631729 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.634715 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fxv\" (UniqueName: \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.634951 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.635932 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:16.636493 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-serving-cert: secret "maas-api-e2e-shared-a-w1-be8b73-serving-cert" not found Oct 30 21:41:16.638437 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:16.636563 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls podName:c345b684-a3f9-4bdf-9d7d-7a22b56067df nodeName:}" failed. No retries permitted until 2026-10-30 21:41:17.136541254 +0000 UTC m=+1791.119241670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls") pod "maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" (UID: "c345b684-a3f9-4bdf-9d7d-7a22b56067df") : secret "maas-api-e2e-shared-a-w1-be8b73-serving-cert" not found Oct 30 21:41:16.661173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.660636 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:16.697193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:16.697160 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fxv\" (UniqueName: \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:17.144591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.144557 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:17.146920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.146901 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:17.303131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.303100 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:17.521491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.521454 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:41:17.521646 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.521515 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:41:17.733240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.730579 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:41:17.766285 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.766246 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:41:17.770773 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.770659 4921 generic.go:356] "Generic (PLEG): container finished" podID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerID="faa2111eecc9e4c3ef9a29c0d465d997a120d23eb52c928407c30c3c384263f0" exitCode=0 Oct 30 21:41:17.771403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.770937 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" event={"ID":"aa26ecea-8202-447f-a30f-74a73ec12f0c","Type":"ContainerDied","Data":"faa2111eecc9e4c3ef9a29c0d465d997a120d23eb52c928407c30c3c384263f0"} Oct 30 21:41:17.773335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.773282 4921 generic.go:356] "Generic (PLEG): container finished" podID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerID="beab3a31d27a8587f2b8fbe0e49f695bc4b1942fed0edd469616fa56c2503858" exitCode=0 Oct 30 21:41:17.773607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.773538 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" event={"ID":"77b71d9c-4cb4-4653-bf8f-802d4ad24e79","Type":"ContainerDied","Data":"beab3a31d27a8587f2b8fbe0e49f695bc4b1942fed0edd469616fa56c2503858"} Oct 30 21:41:17.776412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.776389 4921 generic.go:356] "Generic (PLEG): container finished" podID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerID="20e4e4320fbde60c71abd348c1913366c5c8d7f9b4fa31afb0ea0384c12e38d6" exitCode=0 Oct 30 21:41:17.776508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.776492 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" event={"ID":"048998df-7505-4e39-b56e-6a1f4d131fb3","Type":"ContainerDied","Data":"20e4e4320fbde60c71abd348c1913366c5c8d7f9b4fa31afb0ea0384c12e38d6"} Oct 30 21:41:17.778604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.778579 4921 generic.go:356] "Generic (PLEG): container finished" podID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerID="94e02d49cc0a06a731a452e4f8a222d58e058ff7d74ec57f61c246c0bcc9b64e" exitCode=0 Oct 30 21:41:17.778725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.778707 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" event={"ID":"3b8fbb45-2be4-4716-966b-df3222baccbe","Type":"ContainerDied","Data":"94e02d49cc0a06a731a452e4f8a222d58e058ff7d74ec57f61c246c0bcc9b64e"} Oct 30 21:41:17.782529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.782409 4921 generic.go:356] "Generic (PLEG): container finished" podID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerID="80c9f3c7b7ffac08ae619873c3c74619adf56a1557bd870774a5d3b507ca5622" exitCode=0 Oct 30 21:41:17.782529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.782484 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" event={"ID":"a95637a3-5393-46bc-8cfc-9c30ee489828","Type":"ContainerDied","Data":"80c9f3c7b7ffac08ae619873c3c74619adf56a1557bd870774a5d3b507ca5622"} Oct 30 21:41:17.785863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.784946 4921 generic.go:356] "Generic (PLEG): container finished" podID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerID="2ec0902066a4095e31c473dccba37130cb05dadee3ba375a7a23ac1a7f41d923" exitCode=0 Oct 30 21:41:17.785863 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.785036 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" event={"ID":"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5","Type":"ContainerDied","Data":"2ec0902066a4095e31c473dccba37130cb05dadee3ba375a7a23ac1a7f41d923"} Oct 30 21:41:17.786730 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.786714 4921 generic.go:356] "Generic (PLEG): container finished" podID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerID="8eb80001e2b12bfdd43b840c07db76b806c54242752d4c649bccc8a7e948f5f1" exitCode=0 Oct 30 21:41:17.786911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.786795 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" event={"ID":"2264b3da-491c-44b8-b21e-9a1fecf33f10","Type":"ContainerDied","Data":"8eb80001e2b12bfdd43b840c07db76b806c54242752d4c649bccc8a7e948f5f1"} Oct 30 21:41:17.786984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.786941 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" event={"ID":"2264b3da-491c-44b8-b21e-9a1fecf33f10","Type":"ContainerDied","Data":"7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645"} Oct 30 21:41:17.787118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.786983 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdffff0e118b656eae175d93ed7e12278da760f9d0d723483dc2130498ec645" Oct 30 21:41:17.789533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.789471 4921 generic.go:356] "Generic (PLEG): container finished" podID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerID="f3ca8a9c6502568cbf62655ca97c46e9fea32707cceca28db9636aa3dc87baff" exitCode=0 Oct 30 21:41:17.789624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.789535 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerDied","Data":"f3ca8a9c6502568cbf62655ca97c46e9fea32707cceca28db9636aa3dc87baff"} Oct 30 21:41:17.789624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.789582 4921 scope.go:122] "RemoveContainer" containerID="55573edce635bf993fae5955fb4fef3d9d2ad91814bb3140f62e43f486d6bcfe" Oct 30 21:41:17.793035 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.792813 4921 generic.go:356] "Generic (PLEG): container finished" podID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerID="0e83ec614700c345a315b6b6f94e080e358027a6b2efe5fcc2ef8b3034ac3fb5" exitCode=0 Oct 30 21:41:17.793035 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:17.792934 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerDied","Data":"0e83ec614700c345a315b6b6f94e080e358027a6b2efe5fcc2ef8b3034ac3fb5"} Oct 30 21:41:18.009805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.009782 4921 scope.go:122] "RemoveContainer" containerID="326aeee22586cb838654c115728511bd908cd93b315022aeb1f855879ab592bf" Oct 30 21:41:18.012140 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.011675 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:41:18.018029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.018009 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:41:18.022844 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.022774 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:41:18.056254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.056040 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:41:18.062540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.062500 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:41:18.066830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.066756 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:41:18.101929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.101876 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:41:18.108919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.108860 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:41:18.112585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.112518 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:41:18.167766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.167735 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.168566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.168544 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.168750 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.168735 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.168837 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.168811 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.168913 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.168899 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.169050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169035 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.169142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169120 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.169298 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169161 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.169539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169520 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.169681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169521 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.169758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169657 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.169758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169668 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.169859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169724 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.169859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169790 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.169859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169792 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.169859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169825 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169858 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169918 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169953 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169965 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.169982 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170011 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.170067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170051 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170092 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170121 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170188 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170237 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170265 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170299 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170374 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170403 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170434 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170470 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170499 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170551 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170580 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170603 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170647 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170678 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170708 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170735 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170768 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170803 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170835 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170875 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170906 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.170988 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171053 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171092 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171136 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171188 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171228 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\") pod \"a95637a3-5393-46bc-8cfc-9c30ee489828\" (UID: \"a95637a3-5393-46bc-8cfc-9c30ee489828\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171267 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171298 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\") pod \"aa26ecea-8202-447f-a30f-74a73ec12f0c\" (UID: \"aa26ecea-8202-447f-a30f-74a73ec12f0c\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171356 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171439 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171468 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\") pod \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\" (UID: \"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171532 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\") pod \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\" (UID: \"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171592 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\") pod \"2264b3da-491c-44b8-b21e-9a1fecf33f10\" (UID: \"2264b3da-491c-44b8-b21e-9a1fecf33f10\") " Oct 30 21:41:18.171806 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.171624 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\") pod \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\" (UID: \"c72fa16b-e51e-4a6d-8993-69b9033e8e77\") " Oct 30 21:41:18.172469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.172335 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.172469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.172445 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.172832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.172692 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.172921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.172844 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.173264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173211 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173235 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173251 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173266 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173280 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173295 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173310 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173341 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173357 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173440 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173464 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.173625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173617 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.174097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.173686 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.175046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.175022 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.175747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.175386 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.175747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.175470 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.177936 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.177657 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.178385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.178358 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.178829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.178557 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.179267 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.179483 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.179719 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.179725 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.179799 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.180532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.180024 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.180964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.180723 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.181441 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.181206 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.181441 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.181257 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.187260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.187240 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.188726 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.187730 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.188726 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.188069 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.191383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.191360 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.191607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.191587 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.191698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.191683 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.192978 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.192714 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.192978 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.192731 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.195234 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.195209 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.195508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.195455 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.196497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.196471 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.197604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.197579 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.198851 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.198803 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.198937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.198851 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "kube-api-access-hklqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205682 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205689 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205687 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205761 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205776 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205798 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205825 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q" pod "c72fa16b-e51e-4a6d-8993-69b9033e8e77" (UID: "c72fa16b-e51e-4a6d-8993-69b9033e8e77"). InnerVolumeSpecName "kube-api-access-bld8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205835 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo" pod "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" (UID: "b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205849 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89" pod "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" (UID: "e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9"). InnerVolumeSpecName "kube-api-access-8xl89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205683 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg" pod "aa26ecea-8202-447f-a30f-74a73ec12f0c" (UID: "aa26ecea-8202-447f-a30f-74a73ec12f0c"). InnerVolumeSpecName "kube-api-access-whlmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.205948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.205922 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9" pod "a95637a3-5393-46bc-8cfc-9c30ee489828" (UID: "a95637a3-5393-46bc-8cfc-9c30ee489828"). InnerVolumeSpecName "kube-api-access-sk7g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.207651 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.207625 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k" pod "2264b3da-491c-44b8-b21e-9a1fecf33f10" (UID: "2264b3da-491c-44b8-b21e-9a1fecf33f10"). InnerVolumeSpecName "kube-api-access-nw96k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.220910 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.220833 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:41:18.227246 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.227179 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:41:18.274728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274690 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.274877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274777 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.274877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274807 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.274877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274846 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.274877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274874 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275086 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274943 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275086 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.274969 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275086 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275001 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275086 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275042 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275112 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275139 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275178 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275204 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275238 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275275 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275399 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275427 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275490 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.275716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275548 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275575 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275642 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275698 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.275902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275814 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.275902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275871 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.276003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275917 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\") pod \"048998df-7505-4e39-b56e-6a1f4d131fb3\" (UID: \"048998df-7505-4e39-b56e-6a1f4d131fb3\") " Oct 30 21:41:18.276003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275946 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\") pod \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\" (UID: \"77b71d9c-4cb4-4653-bf8f-802d4ad24e79\") " Oct 30 21:41:18.276003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.275985 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\") pod \"3b8fbb45-2be4-4716-966b-df3222baccbe\" (UID: \"3b8fbb45-2be4-4716-966b-df3222baccbe\") " Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.276630 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.276746 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.276969 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.276991 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277006 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277021 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277035 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277049 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa26ecea-8202-447f-a30f-74a73ec12f0c-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277064 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277078 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277091 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277103 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277150 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277165 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bld8q\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-kube-api-access-bld8q\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277179 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277193 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277207 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277221 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277235 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hklqd\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-kube-api-access-hklqd\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277249 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277264 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2264b3da-491c-44b8-b21e-9a1fecf33f10-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277279 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xl89\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-kube-api-access-8xl89\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277296 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95637a3-5393-46bc-8cfc-9c30ee489828-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277311 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277351 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whlmg\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-kube-api-access-whlmg\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277365 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw96k\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-kube-api-access-nw96k\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277379 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277393 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277406 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277420 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277433 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277447 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277462 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa26ecea-8202-447f-a30f-74a73ec12f0c-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277477 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277490 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.277491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277506 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277520 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277536 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa26ecea-8202-447f-a30f-74a73ec12f0c-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277551 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2264b3da-491c-44b8-b21e-9a1fecf33f10-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277565 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277579 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7g9\" (UniqueName: \"kubernetes.io/projected/a95637a3-5393-46bc-8cfc-9c30ee489828-kube-api-access-sk7g9\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277594 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277610 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277625 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95637a3-5393-46bc-8cfc-9c30ee489828-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277639 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2264b3da-491c-44b8-b21e-9a1fecf33f10-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277654 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72fa16b-e51e-4a6d-8993-69b9033e8e77-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277669 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.277685 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.278164 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.278218 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.278528 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.278911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.278667 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.279435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.278923 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.279435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279015 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "kube-api-access-snfwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.279435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279068 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.279435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279273 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.279435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279363 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.279691 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279668 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.279954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279910 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.280050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.279959 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.280115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.280081 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.280799 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.280500 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.280951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.280935 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.286349 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.286299 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.286530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.286506 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.293924 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.293904 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.294039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.294025 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "kube-api-access-9ctd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.294207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.294182 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.299478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.299426 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.302479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.302435 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.306287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.305498 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo" pod "048998df-7505-4e39-b56e-6a1f4d131fb3" (UID: "048998df-7505-4e39-b56e-6a1f4d131fb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.313200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.308159 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo" pod "77b71d9c-4cb4-4653-bf8f-802d4ad24e79" (UID: "77b71d9c-4cb4-4653-bf8f-802d4ad24e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.316406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.316379 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj" pod "3b8fbb45-2be4-4716-966b-df3222baccbe" (UID: "3b8fbb45-2be4-4716-966b-df3222baccbe"). InnerVolumeSpecName "kube-api-access-m4tmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.378383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378360 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.378479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378399 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.378479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378449 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.378687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378488 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.378687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378511 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.378687 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378552 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.378875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378839 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.378937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378884 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.379002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378939 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.379002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378969 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.379002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.378981 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.379162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379052 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.379162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379099 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.379162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379112 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.379551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379485 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.379551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379507 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379623 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379688 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379775 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379811 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379856 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379909 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\") pod \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\" (UID: \"b25a12b3-893c-43af-9e24-ff2e3f4b84c8\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.379986 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\") pod \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\" (UID: \"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2\") " Oct 30 21:41:18.380519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380399 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380558 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380699 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380812 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380908 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380927 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380948 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.380968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380964 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctd4\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-kube-api-access-9ctd4\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380978 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.380992 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381007 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381020 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381033 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381046 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381061 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381074 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381086 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b8fbb45-2be4-4716-966b-df3222baccbe-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381099 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381111 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381124 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381136 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381149 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381161 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048998df-7505-4e39-b56e-6a1f4d131fb3-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381173 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381185 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381199 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048998df-7505-4e39-b56e-6a1f4d131fb3-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381212 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381227 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381240 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381253 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381266 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b8fbb45-2be4-4716-966b-df3222baccbe-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381280 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381293 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381306 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snfwq\" (UniqueName: \"kubernetes.io/projected/048998df-7505-4e39-b56e-6a1f4d131fb3-kube-api-access-snfwq\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381334 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77b71d9c-4cb4-4653-bf8f-802d4ad24e79-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381347 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.381348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381362 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4tmj\" (UniqueName: \"kubernetes.io/projected/3b8fbb45-2be4-4716-966b-df3222baccbe-kube-api-access-m4tmj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381375 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b8fbb45-2be4-4716-966b-df3222baccbe-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.381795 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.382361 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.382392 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "kube-api-access-5hr4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.382655 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo" pod "b25a12b3-893c-43af-9e24-ff2e3f4b84c8" (UID: "b25a12b3-893c-43af-9e24-ff2e3f4b84c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.383036 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:18.383269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.383246 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "kube-api-access-wslws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.383692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.383672 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:41:18.383772 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.383720 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token" pod "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" (UID: "2e7ee2a4-3c61-4a37-94d0-caa9b55aead2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:18.483245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483210 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr4q\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-kube-api-access-5hr4q\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483250 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483266 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483279 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wslws\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-kube-api-access-wslws\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483293 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483306 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483364 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483376 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483392 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.483422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.483405 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25a12b3-893c-43af-9e24-ff2e3f4b84c8-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:18.820774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.820693 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" event={"ID":"c345b684-a3f9-4bdf-9d7d-7a22b56067df","Type":"ContainerStarted","Data":"c0377ef60c459276d91d034949c30d2091f6ed6b95b64b14a4e31c801d670695"} Oct 30 21:41:18.822354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.822292 4921 generic.go:356] "Generic (PLEG): container finished" podID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerID="bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d" exitCode=0 Oct 30 21:41:18.822484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.822423 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" event={"ID":"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2","Type":"ContainerDied","Data":"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d"} Oct 30 21:41:18.822484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.822470 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" event={"ID":"2e7ee2a4-3c61-4a37-94d0-caa9b55aead2","Type":"ContainerDied","Data":"d988a4382ab744d1aee61959e5f86670c16cfe213f2a43936331b234a2d7d078"} Oct 30 21:41:18.822656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.822503 4921 scope.go:122] "RemoveContainer" containerID="bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d" Oct 30 21:41:18.822656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.822625 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp" Oct 30 21:41:18.834207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.834187 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" event={"ID":"aa26ecea-8202-447f-a30f-74a73ec12f0c","Type":"ContainerDied","Data":"feabf35bb1e2ffa53586fc8ded884ea106f6fb292f1544bb0ac5d2b524591e0e"} Oct 30 21:41:18.834598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.834585 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m" Oct 30 21:41:18.836919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.836796 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" event={"ID":"77b71d9c-4cb4-4653-bf8f-802d4ad24e79","Type":"ContainerDied","Data":"a5e4e60816718062fd10125af816aa612c5e1e3c665f729966d682df8d9dd2c5"} Oct 30 21:41:18.836919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.836878 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86" Oct 30 21:41:18.839774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.839755 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" event={"ID":"048998df-7505-4e39-b56e-6a1f4d131fb3","Type":"ContainerDied","Data":"f84b451640128f481bfb45ce2b20366342c633d78a08bb9769ce780305e36d1d"} Oct 30 21:41:18.839969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.839956 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8" Oct 30 21:41:18.854681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.854659 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" event={"ID":"3b8fbb45-2be4-4716-966b-df3222baccbe","Type":"ContainerDied","Data":"29f89f7062786538363d08d13fd8368f3c31ff293fd41f1f569904425a97c336"} Oct 30 21:41:18.854871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.854856 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn" Oct 30 21:41:18.860439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.858579 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" event={"ID":"a95637a3-5393-46bc-8cfc-9c30ee489828","Type":"ContainerDied","Data":"d5b0a88d1338a722e85cb613a6dea5de02324de09d356874731e328d72423cbc"} Oct 30 21:41:18.860439 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.858657 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h" Oct 30 21:41:18.876537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.875837 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" event={"ID":"b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5","Type":"ContainerDied","Data":"6808b870164994dfaab031926b4c554ff78fd66b3382e6162ef7a7111359772d"} Oct 30 21:41:18.876537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.875903 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2" Oct 30 21:41:18.878146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.878129 4921 scope.go:122] "RemoveContainer" containerID="bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d" Oct 30 21:41:18.878575 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:18.878548 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d\": container with ID starting with bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d not found: ID does not exist" containerID="bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d" Oct 30 21:41:18.878642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.878616 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d"} err="failed to get container status \"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d\": rpc error: code = NotFound desc = could not find container \"bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d\": container with ID starting with bad681fdbc27717504f230613278993acedeefc0373084ad1164d0c0bca3350d not found: ID does not exist" Oct 30 21:41:18.878682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.878642 4921 scope.go:122] "RemoveContainer" containerID="faa2111eecc9e4c3ef9a29c0d465d997a120d23eb52c928407c30c3c384263f0" Oct 30 21:41:18.879292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.878914 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" event={"ID":"e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9","Type":"ContainerDied","Data":"d1f61b03ebfd256dbee88e0c1c3c225f951fdcd907339551c715ce706eb4b2d9"} Oct 30 21:41:18.879292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.879072 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m" Oct 30 21:41:18.881419 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.881395 4921 generic.go:356] "Generic (PLEG): container finished" podID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerID="1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2" exitCode=0 Oct 30 21:41:18.881634 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.881508 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" Oct 30 21:41:18.882998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.882985 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m" Oct 30 21:41:18.886819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.885422 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" Oct 30 21:41:18.886819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.886485 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" event={"ID":"b25a12b3-893c-43af-9e24-ff2e3f4b84c8","Type":"ContainerDied","Data":"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2"} Oct 30 21:41:18.886819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.886527 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b" event={"ID":"b25a12b3-893c-43af-9e24-ff2e3f4b84c8","Type":"ContainerDied","Data":"ce900d6c901244245fab38c72bbecb95305945bb8ff7fcf83f6b9022a3c32969"} Oct 30 21:41:18.886819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.886552 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl" event={"ID":"c72fa16b-e51e-4a6d-8993-69b9033e8e77","Type":"ContainerDied","Data":"77ffd041dabbce71b03f4e0c33ee19fc0b62223fc284b5b6506847d12931d19c"} Oct 30 21:41:18.903864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.903845 4921 scope.go:122] "RemoveContainer" containerID="beab3a31d27a8587f2b8fbe0e49f695bc4b1942fed0edd469616fa56c2503858" Oct 30 21:41:18.917175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.917156 4921 scope.go:122] "RemoveContainer" containerID="20e4e4320fbde60c71abd348c1913366c5c8d7f9b4fa31afb0ea0384c12e38d6" Oct 30 21:41:18.930833 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.930475 4921 scope.go:122] "RemoveContainer" containerID="94e02d49cc0a06a731a452e4f8a222d58e058ff7d74ec57f61c246c0bcc9b64e" Oct 30 21:41:18.961616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.961578 4921 scope.go:122] "RemoveContainer" containerID="80c9f3c7b7ffac08ae619873c3c74619adf56a1557bd870774a5d3b507ca5622" Oct 30 21:41:18.981126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.981106 4921 scope.go:122] "RemoveContainer" containerID="2ec0902066a4095e31c473dccba37130cb05dadee3ba375a7a23ac1a7f41d923" Oct 30 21:41:18.995366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:18.995340 4921 scope.go:122] "RemoveContainer" containerID="f3ca8a9c6502568cbf62655ca97c46e9fea32707cceca28db9636aa3dc87baff" Oct 30 21:41:19.005558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.005536 4921 scope.go:122] "RemoveContainer" containerID="1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2" Oct 30 21:41:19.022513 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.018171 4921 scope.go:122] "RemoveContainer" containerID="1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2" Oct 30 21:41:19.022513 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:19.018539 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2\": container with ID starting with 1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2 not found: ID does not exist" containerID="1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2" Oct 30 21:41:19.022513 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.018572 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2"} err="failed to get container status \"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2\": rpc error: code = NotFound desc = could not find container \"1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2\": container with ID starting with 1c9a8722c80335eb7265903b75b88c0db346e099c154786abe1c27680d5fdbb2 not found: ID does not exist" Oct 30 21:41:19.022513 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.018596 4921 scope.go:122] "RemoveContainer" containerID="0e83ec614700c345a315b6b6f94e080e358027a6b2efe5fcc2ef8b3034ac3fb5" Oct 30 21:41:19.403225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.403123 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:41:19.409416 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.409386 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-4sd7b"] Oct 30 21:41:19.514230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.514191 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:41:19.524422 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.524346 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-mk89m"] Oct 30 21:41:19.630508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.630474 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:41:19.634729 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.634706 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-49d86"] Oct 30 21:41:19.760366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.758366 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:41:19.766231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.766201 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-fzkd8"] Oct 30 21:41:19.882852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.882818 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:41:19.886877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.886849 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-s6z6m"] Oct 30 21:41:19.895169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.895145 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" event={"ID":"c345b684-a3f9-4bdf-9d7d-7a22b56067df","Type":"ContainerStarted","Data":"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208"} Oct 30 21:41:19.895300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.895266 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:19.995885 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:19.995855 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:41:20.010277 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.010248 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-78f5m"] Oct 30 21:41:20.121956 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.121886 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:41:20.130928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.130905 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-mxrfn"] Oct 30 21:41:20.169208 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.169183 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" path="/var/lib/kubelet/pods/048998df-7505-4e39-b56e-6a1f4d131fb3/volumes" Oct 30 21:41:20.169732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.169716 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" path="/var/lib/kubelet/pods/2264b3da-491c-44b8-b21e-9a1fecf33f10/volumes" Oct 30 21:41:20.170188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.170175 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" path="/var/lib/kubelet/pods/3b8fbb45-2be4-4716-966b-df3222baccbe/volumes" Oct 30 21:41:20.170960 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.170945 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" path="/var/lib/kubelet/pods/77b71d9c-4cb4-4653-bf8f-802d4ad24e79/volumes" Oct 30 21:41:20.171411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.171398 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" path="/var/lib/kubelet/pods/aa26ecea-8202-447f-a30f-74a73ec12f0c/volumes" Oct 30 21:41:20.171829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.171817 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" path="/var/lib/kubelet/pods/b25a12b3-893c-43af-9e24-ff2e3f4b84c8/volumes" Oct 30 21:41:20.172533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.172520 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" path="/var/lib/kubelet/pods/e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9/volumes" Oct 30 21:41:20.322177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.322149 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:41:20.326388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.326367 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-9vxsl"] Oct 30 21:41:20.437633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.437576 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:41:20.445162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.445136 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f52c3-openshift-default-df5ffd65f-kch5h"] Oct 30 21:41:20.558475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.558449 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:41:20.562080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.562060 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-be9451-openshift-default-565fb88cd9-vkwb2"] Oct 30 21:41:20.742821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.742785 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:41:20.746762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.746733 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-2pxwp"] Oct 30 21:41:20.872695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:20.872601 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" podStartSLOduration=3.910184609 podStartE2EDuration="4.872588575s" totalImagesPullingTime="962.403966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:20.866477058 +0000 UTC m=+1794.849177478" watchObservedRunningTime="2026-10-30 21:41:20.872588575 +0000 UTC m=+1794.855288993" Oct 30 21:41:22.170696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:22.170666 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" path="/var/lib/kubelet/pods/2e7ee2a4-3c61-4a37-94d0-caa9b55aead2/volumes" Oct 30 21:41:22.171136 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:22.171121 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" path="/var/lib/kubelet/pods/a95637a3-5393-46bc-8cfc-9c30ee489828/volumes" Oct 30 21:41:22.171566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:22.171553 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" path="/var/lib/kubelet/pods/b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5/volumes" Oct 30 21:41:22.172286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:22.172271 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" path="/var/lib/kubelet/pods/c72fa16b-e51e-4a6d-8993-69b9033e8e77/volumes" Oct 30 21:41:25.921053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:25.921023 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:41:26.470232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470190 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:41:26.470852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470834 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.470852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470850 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470869 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470874 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470885 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470890 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470896 4921 cpu_manager.go:414] "containerMap: removing container" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470901 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470910 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470914 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470924 4921 cpu_manager.go:414] "containerMap: removing container" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470929 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470939 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470943 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470950 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470954 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470962 4921 cpu_manager.go:414] "containerMap: removing container" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470967 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470974 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470978 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470985 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" Oct 30 21:41:26.470987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.470990 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471013 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471018 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471028 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471033 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471110 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b430dd5c-6f2d-4034-8d7b-08f78e3ddeb5" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471119 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471126 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8fbb45-2be4-4716-966b-df3222baccbe" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471132 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2264b3da-491c-44b8-b21e-9a1fecf33f10" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471139 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b71d9c-4cb4-4653-bf8f-802d4ad24e79" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471144 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bb6d9b-27e0-4b71-be79-c1c66cba0fb9" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471150 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="048998df-7505-4e39-b56e-6a1f4d131fb3" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471156 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471160 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95637a3-5393-46bc-8cfc-9c30ee489828" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471165 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa26ecea-8202-447f-a30f-74a73ec12f0c" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471172 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72fa16b-e51e-4a6d-8993-69b9033e8e77" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471178 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e7ee2a4-3c61-4a37-94d0-caa9b55aead2" containerName="istio-proxy" Oct 30 21:41:26.471599 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.471186 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25a12b3-893c-43af-9e24-ff2e3f4b84c8" containerName="istio-proxy" Oct 30 21:41:26.476421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.476401 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.532451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.532424 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7cf226-serving-cert\"" Oct 30 21:41:26.535822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.535793 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:41:26.604388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.604346 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.604690 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.604669 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.604815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.604798 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5sp\" (UniqueName: \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.605130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.605105 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.688213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.688185 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:41:26.700016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.699992 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:41:26.700193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.700052 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:41:26.707214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.706604 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.707214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.706693 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.707214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.706730 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5sp\" (UniqueName: \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.707214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.706946 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.707917 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:26.707895 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-serving-cert: secret "maas-api-e2e-shared-a-w3-7cf226-serving-cert" not found Oct 30 21:41:26.708026 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:26.707965 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls podName:11ca0026-ae00-428e-82ce-b92bfaa8db95 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:27.207943228 +0000 UTC m=+1801.190643625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls") pod "maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" (UID: "11ca0026-ae00-428e-82ce-b92bfaa8db95") : secret "maas-api-e2e-shared-a-w3-7cf226-serving-cert" not found Oct 30 21:41:26.709181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.709153 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.710230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.710210 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.768399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.768254 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:41:26.775405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.775069 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5sp\" (UniqueName: \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:26.843836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.843807 4921 scope.go:122] "RemoveContainer" containerID="7dcc0350b6028bcc03cfb78361cf1884fcf46f640c1416050bd77b3b90f94e55" Oct 30 21:41:26.858803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.858776 4921 scope.go:122] "RemoveContainer" containerID="a18c6937e759184c692fcecc347bafff28ac746941cb2410c838e4968f55a740" Oct 30 21:41:26.884240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.884219 4921 scope.go:122] "RemoveContainer" containerID="8eb80001e2b12bfdd43b840c07db76b806c54242752d4c649bccc8a7e948f5f1" Oct 30 21:41:26.897130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:26.897110 4921 scope.go:122] "RemoveContainer" containerID="05690b1d60db9a9364fc81ad38d6f1c5f15ea9d2cacefc29b67fa7df69145b97" Oct 30 21:41:27.215729 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:27.215697 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:27.217980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:27.217955 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:27.391079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:27.391004 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:27.826180 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:41:27.826146 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ca0026_ae00_428e_82ce_b92bfaa8db95.slice/crio-8fefd691be071516bd8265dd53dd0b7faf318cf0473ab9364243c7258df3f639 WatchSource:0}: Error finding container 8fefd691be071516bd8265dd53dd0b7faf318cf0473ab9364243c7258df3f639: Status 404 returned error can't find the container with id 8fefd691be071516bd8265dd53dd0b7faf318cf0473ab9364243c7258df3f639 Oct 30 21:41:27.829374 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:27.829345 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:41:27.978563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:27.978526 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" event={"ID":"11ca0026-ae00-428e-82ce-b92bfaa8db95","Type":"ContainerStarted","Data":"8fefd691be071516bd8265dd53dd0b7faf318cf0473ab9364243c7258df3f639"} Oct 30 21:41:28.987638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:28.987604 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" event={"ID":"11ca0026-ae00-428e-82ce-b92bfaa8db95","Type":"ContainerStarted","Data":"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5"} Oct 30 21:41:28.988103 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:28.987687 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:29.113333 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:29.113269 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" podStartSLOduration=2.3439135159999998 podStartE2EDuration="3.113254238s" totalImagesPullingTime="769.340722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:29.108922938 +0000 UTC m=+1803.091623364" watchObservedRunningTime="2026-10-30 21:41:29.113254238 +0000 UTC m=+1803.095954656" Oct 30 21:41:32.250391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:32.250357 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:41:32.300665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:32.300630 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:41:35.001245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:35.001218 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:41:36.253189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.253153 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:41:36.257767 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.257743 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.268866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.268839 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.269054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.269026 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc62\" (UniqueName: \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.269150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.269131 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.269279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.269264 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.312377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.312297 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-815dab-serving-cert\"" Oct 30 21:41:36.366326 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.366282 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:41:36.370501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.370476 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.370662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.370645 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc62\" (UniqueName: \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.370734 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.370695 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.371090 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:36.370852 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-serving-cert: secret "maas-api-e2e-shared-a-w2-815dab-serving-cert" not found Oct 30 21:41:36.371090 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:36.370926 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls podName:c56e56e0-e586-4f59-afc2-7c10885a6535 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:36.870905786 +0000 UTC m=+1810.853606183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls") pod "maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" (UID: "c56e56e0-e586-4f59-afc2-7c10885a6535") : secret "maas-api-e2e-shared-a-w2-815dab-serving-cert" not found Oct 30 21:41:36.371706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.371301 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.371963 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.371413 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.376508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.376488 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.628921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.627381 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc62\" (UniqueName: \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.928138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.928062 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:36.930591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:36.930564 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:37.173337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:37.172804 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:37.541581 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:37.541538 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:41:37.569373 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:41:37.569307 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56e56e0_e586_4f59_afc2_7c10885a6535.slice/crio-7de87a3371640286c3745e9d92c684fe26cae2379dc9f3348a352bbab036a641 WatchSource:0}: Error finding container 7de87a3371640286c3745e9d92c684fe26cae2379dc9f3348a352bbab036a641: Status 404 returned error can't find the container with id 7de87a3371640286c3745e9d92c684fe26cae2379dc9f3348a352bbab036a641 Oct 30 21:41:37.570493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:37.570470 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:41:38.118289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:38.118255 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" event={"ID":"c56e56e0-e586-4f59-afc2-7c10885a6535","Type":"ContainerStarted","Data":"7de87a3371640286c3745e9d92c684fe26cae2379dc9f3348a352bbab036a641"} Oct 30 21:41:39.130588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:39.130546 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" event={"ID":"c56e56e0-e586-4f59-afc2-7c10885a6535","Type":"ContainerStarted","Data":"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d"} Oct 30 21:41:39.131007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:39.130674 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:39.247390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:39.247306 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" podStartSLOduration=2.403295731 podStartE2EDuration="3.247292545s" totalImagesPullingTime="843.996814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:39.245470749 +0000 UTC m=+1813.228171167" watchObservedRunningTime="2026-10-30 21:41:39.247292545 +0000 UTC m=+1813.229992980" Oct 30 21:41:45.143828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:45.143799 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:41:47.271062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.271005 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" containerID="cri-o://8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24" gracePeriod=45 Oct 30 21:41:47.322900 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.322831 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" containerID="cri-o://98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b" gracePeriod=45 Oct 30 21:41:47.632572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.632501 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:41:47.652353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.651858 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:41:47.771841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.771805 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.771885 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\" (UniqueName: \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.771926 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772028 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772099 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772115 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.772309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772255 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772271 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.772309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772332 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772366 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\") pod \"2089674d-33e5-406a-8f53-55ece03fe7ef\" (UID: \"2089674d-33e5-406a-8f53-55ece03fe7ef\") " Oct 30 21:41:47.772523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772398 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772443 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\" (UniqueName: \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772478 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\" (UniqueName: \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772527 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\") pod \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\" (UID: \"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7\") " Oct 30 21:41:47.772735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772658 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.772855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.772835 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.773132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.773109 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.773240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.773136 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.773240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.773148 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.773240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.773156 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.775615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.775595 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.775754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.775739 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.777428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.777405 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "kube-api-access-77jsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:47.777969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.777943 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:47.778636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.778611 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.778736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.778713 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm" pod "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" (UID: "7369cab6-4366-42bb-8b8e-dfef7e7f9bc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:47.778736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.778714 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "kube-api-access-qf8pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:47.779953 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.779930 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs" pod "2089674d-33e5-406a-8f53-55ece03fe7ef" (UID: "2089674d-33e5-406a-8f53-55ece03fe7ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:47.875976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875889 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8pm\" (UniqueName: \"kubernetes.io/projected/2089674d-33e5-406a-8f53-55ece03fe7ef-kube-api-access-qf8pm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.875976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875933 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.875976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875948 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2089674d-33e5-406a-8f53-55ece03fe7ef-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.875976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875961 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2089674d-33e5-406a-8f53-55ece03fe7ef-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.875976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875973 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.876369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.875985 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.876369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.876000 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jsr\" (UniqueName: \"kubernetes.io/projected/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-kube-api-access-77jsr\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:47.876369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:47.876012 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:41:48.203025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.202992 4921 generic.go:356] "Generic (PLEG): container finished" podID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerID="8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24" exitCode=0 Oct 30 21:41:48.203157 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.203104 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" Oct 30 21:41:48.203232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.203150 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" event={"ID":"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7","Type":"ContainerDied","Data":"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24"} Oct 30 21:41:48.203232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.203184 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7" event={"ID":"7369cab6-4366-42bb-8b8e-dfef7e7f9bc7","Type":"ContainerDied","Data":"1668444ef18ba1d1efebadd689d4ce5c9ba338f51ca4e77210a3ebdd5d5bba4c"} Oct 30 21:41:48.203232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.203201 4921 scope.go:122] "RemoveContainer" containerID="8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24" Oct 30 21:41:48.205930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.205904 4921 generic.go:356] "Generic (PLEG): container finished" podID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerID="98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b" exitCode=0 Oct 30 21:41:48.206043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.206007 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" event={"ID":"2089674d-33e5-406a-8f53-55ece03fe7ef","Type":"ContainerDied","Data":"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b"} Oct 30 21:41:48.206096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.206038 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" event={"ID":"2089674d-33e5-406a-8f53-55ece03fe7ef","Type":"ContainerDied","Data":"070b5219c599a86d69e99b8a33c5e1f3656459a678fa6ded0bc9e1d4e5ee543b"} Oct 30 21:41:48.206133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.206099 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875" Oct 30 21:41:48.228425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.228406 4921 scope.go:122] "RemoveContainer" containerID="8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24" Oct 30 21:41:48.228741 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:48.228722 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24\": container with ID starting with 8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24 not found: ID does not exist" containerID="8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24" Oct 30 21:41:48.228845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.228750 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24"} err="failed to get container status \"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24\": rpc error: code = NotFound desc = could not find container \"8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24\": container with ID starting with 8dbf7af10429a05f2d427094ff37e11b8e6e689e8e9b4ef81b7f95ebaad43c24 not found: ID does not exist" Oct 30 21:41:48.228845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.228772 4921 scope.go:122] "RemoveContainer" containerID="98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b" Oct 30 21:41:48.242572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.242524 4921 scope.go:122] "RemoveContainer" containerID="98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b" Oct 30 21:41:48.243143 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:41:48.243116 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b\": container with ID starting with 98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b not found: ID does not exist" containerID="98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b" Oct 30 21:41:48.243239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.243167 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b"} err="failed to get container status \"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b\": rpc error: code = NotFound desc = could not find container \"98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b\": container with ID starting with 98bb8abe5080b8da67694d7a68e7ece58ad06b7611ab0ba9930ed081ba23ae7b not found: ID does not exist" Oct 30 21:41:48.374357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.374293 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:41:48.423073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.423041 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/distinct-w6-9ac5c5-kserve-5db64ccf49-l2ql7"] Oct 30 21:41:48.484300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.484265 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:41:48.488736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:48.488710 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9ac5c5/premium-w6-9ac5c5-kserve-ff547468b-nt875"] Oct 30 21:41:50.172843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:50.172810 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" path="/var/lib/kubelet/pods/2089674d-33e5-406a-8f53-55ece03fe7ef/volumes" Oct 30 21:41:50.173388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:50.173310 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" path="/var/lib/kubelet/pods/7369cab6-4366-42bb-8b8e-dfef7e7f9bc7/volumes" Oct 30 21:41:57.308629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.308586 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:41:57.309213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.308925 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" containerID="cri-o://232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909" gracePeriod=30 Oct 30 21:41:57.356501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.356474 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:41:57.356703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.356668 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" containerID="cri-o://44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f" gracePeriod=30 Oct 30 21:41:57.361104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.361073 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:41:57.361292 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.361260 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" containerID="cri-o://bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c" gracePeriod=30 Oct 30 21:41:57.404025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.403994 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:41:57.404259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.404213 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" containerID="cri-o://201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623" gracePeriod=30 Oct 30 21:41:57.409788 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.409763 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:41:57.409965 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:57.409945 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" containerID="cri-o://d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a" gracePeriod=30 Oct 30 21:41:59.531698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:59.531665 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:41:59.532137 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:41:59.531719 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:02.661510 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.659156 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:42:02.787905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.787872 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.787931 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.787988 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788035 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788125 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788168 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788282 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.788759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788730 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.788958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.788939 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.789307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.789286 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.789484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.789457 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:02.790152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.789900 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.790152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.790122 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.790444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.790221 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\") pod \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\" (UID: \"586eea25-2b06-4402-9c43-ebfe2bbe5caa\") " Oct 30 21:42:02.791175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791151 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.791262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791182 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.791262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791197 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.791262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791210 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.791262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791225 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.791944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.791863 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:42:02.793697 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.793659 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:02.794279 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.794248 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.795961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.795940 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "kube-api-access-frpmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.802303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.802278 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token" pod "586eea25-2b06-4402-9c43-ebfe2bbe5caa" (UID: "586eea25-2b06-4402-9c43-ebfe2bbe5caa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.827068 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.827002 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:42:02.853556 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.853532 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:42:02.856460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.856441 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:42:02.892040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.891982 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892013 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892080 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892109 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892144 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892235 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892254 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.892394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892263 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892358 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892384 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892390 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892419 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892436 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892470 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892491 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892534 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\") pod \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\" (UID: \"6bc2134f-56c5-49af-a0dc-97dc539d20e2\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892560 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.892663 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892640 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.893065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892676 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.893065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.892717 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\") pod \"701aacc1-673e-4302-9bc3-a60d8fdd4405\" (UID: \"701aacc1-673e-4302-9bc3-a60d8fdd4405\") " Oct 30 21:42:02.893161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.893065 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:02.893474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.893273 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.893586 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.893620 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894028 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701aacc1-673e-4302-9bc3-a60d8fdd4405-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894052 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894067 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894080 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894093 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894106 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894119 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894133 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpmj\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-kube-api-access-frpmj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894147 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894160 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586eea25-2b06-4402-9c43-ebfe2bbe5caa-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.894624 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894448 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.895421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894661 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.895421 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.894837 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.896242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.895984 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "kube-api-access-w9q7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.896242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.895999 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.896242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.896212 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.896815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.896790 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.896908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.896825 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.896908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.896869 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo" pod "701aacc1-673e-4302-9bc3-a60d8fdd4405" (UID: "701aacc1-673e-4302-9bc3-a60d8fdd4405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:02.897033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.896987 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "kube-api-access-g4s9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.897468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.897452 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:02.897779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.897758 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy" pod "6bc2134f-56c5-49af-a0dc-97dc539d20e2" (UID: "6bc2134f-56c5-49af-a0dc-97dc539d20e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.994626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994593 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.994815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994691 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.994815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994786 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.994952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994819 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.994952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994853 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.994952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994891 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.994952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994906 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.994952 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994947 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.994982 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995032 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995082 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995117 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995131 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.995181 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995155 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995207 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995233 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995292 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995303 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.995423 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995410 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\") pod \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\" (UID: \"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb\") " Oct 30 21:42:02.995676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995443 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995482 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\") pod \"006d8097-44b3-48d2-85f8-53ff8940234f\" (UID: \"006d8097-44b3-48d2-85f8-53ff8940234f\") " Oct 30 21:42:02.995676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995635 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:02.995927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.995905 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.996109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.996089 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.996284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.996265 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.996971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.996926 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:02.997089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.996992 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.997360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.997332 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:02.997500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.997478 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:02.998142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998126 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998143 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998152 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998162 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9q7v\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-kube-api-access-w9q7v\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998170 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998180 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4s9d\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-kube-api-access-g4s9d\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998188 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998196 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998205 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998213 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998222 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998229 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998237 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998246 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998253 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998261 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc2134f-56c5-49af-a0dc-97dc539d20e2-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998269 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998277 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006d8097-44b3-48d2-85f8-53ff8940234f-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998285 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998293 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998303 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701aacc1-673e-4302-9bc3-a60d8fdd4405-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998331 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701aacc1-673e-4302-9bc3-a60d8fdd4405-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:02.998305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:02.998342 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.000529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.000271 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "kube-api-access-hcrsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:03.001126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.001101 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:03.001200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.001103 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "kube-api-access-gqn7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:03.001200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.001181 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:03.001938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.001913 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:03.002229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.002208 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token" pod "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" (UID: "d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:03.002929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.002905 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo" pod "006d8097-44b3-48d2-85f8-53ff8940234f" (UID: "006d8097-44b3-48d2-85f8-53ff8940234f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:03.100424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100387 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100423 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100438 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrsw\" (UniqueName: \"kubernetes.io/projected/006d8097-44b3-48d2-85f8-53ff8940234f-kube-api-access-hcrsw\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100454 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100467 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqn7r\" (UniqueName: \"kubernetes.io/projected/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb-kube-api-access-gqn7r\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100482 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006d8097-44b3-48d2-85f8-53ff8940234f-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.100666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.100495 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006d8097-44b3-48d2-85f8-53ff8940234f-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:03.366019 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.365991 4921 generic.go:356] "Generic (PLEG): container finished" podID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerID="232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909" exitCode=0 Oct 30 21:42:03.366166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.366044 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" Oct 30 21:42:03.366166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.366045 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" event={"ID":"586eea25-2b06-4402-9c43-ebfe2bbe5caa","Type":"ContainerDied","Data":"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909"} Oct 30 21:42:03.366166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.366111 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" event={"ID":"586eea25-2b06-4402-9c43-ebfe2bbe5caa","Type":"ContainerDied","Data":"78f2690cbfcb4b7b1e1ebc02a2252fe7a01232b169ce1e71c8ed3546990fb851"} Oct 30 21:42:03.366166 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.366141 4921 scope.go:122] "RemoveContainer" containerID="232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909" Oct 30 21:42:03.367951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.367928 4921 generic.go:356] "Generic (PLEG): container finished" podID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerID="bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c" exitCode=0 Oct 30 21:42:03.368062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.367962 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" Oct 30 21:42:03.368062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.367999 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" event={"ID":"701aacc1-673e-4302-9bc3-a60d8fdd4405","Type":"ContainerDied","Data":"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c"} Oct 30 21:42:03.368062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.368041 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j" event={"ID":"701aacc1-673e-4302-9bc3-a60d8fdd4405","Type":"ContainerDied","Data":"ebcfe32ac44d7977d5d11c2751743485af3435a0b725a54e51f1b06fd47c7aec"} Oct 30 21:42:03.369631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.369611 4921 generic.go:356] "Generic (PLEG): container finished" podID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerID="201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623" exitCode=0 Oct 30 21:42:03.369735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.369669 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" Oct 30 21:42:03.369804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.369734 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" event={"ID":"6bc2134f-56c5-49af-a0dc-97dc539d20e2","Type":"ContainerDied","Data":"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623"} Oct 30 21:42:03.369804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.369776 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl" event={"ID":"6bc2134f-56c5-49af-a0dc-97dc539d20e2","Type":"ContainerDied","Data":"f114938bd240e33c6f6574ce3246a1d83aea1e1c508c9229a4bda78f19c31042"} Oct 30 21:42:03.372043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.371456 4921 generic.go:356] "Generic (PLEG): container finished" podID="006d8097-44b3-48d2-85f8-53ff8940234f" containerID="d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a" exitCode=0 Oct 30 21:42:03.372043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.371510 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" Oct 30 21:42:03.372043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.371573 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" event={"ID":"006d8097-44b3-48d2-85f8-53ff8940234f","Type":"ContainerDied","Data":"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a"} Oct 30 21:42:03.372043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.371611 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd" event={"ID":"006d8097-44b3-48d2-85f8-53ff8940234f","Type":"ContainerDied","Data":"e7e18ac2e1d54dbcdf28d8ca96149482a60a3cecd654ecd33fb18c77197cfebd"} Oct 30 21:42:03.373278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.373258 4921 generic.go:356] "Generic (PLEG): container finished" podID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerID="44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f" exitCode=0 Oct 30 21:42:03.373410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.373290 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" event={"ID":"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb","Type":"ContainerDied","Data":"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f"} Oct 30 21:42:03.373410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.373363 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" event={"ID":"d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb","Type":"ContainerDied","Data":"f61c8dea40ca0237df96b9b8860795f9ef75e4d906dac41102dcf2fcaeb808e2"} Oct 30 21:42:03.373530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.373430 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm" Oct 30 21:42:03.394171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.394155 4921 scope.go:122] "RemoveContainer" containerID="232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909" Oct 30 21:42:03.394628 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:03.394508 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909\": container with ID starting with 232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909 not found: ID does not exist" containerID="232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909" Oct 30 21:42:03.394628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.394543 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909"} err="failed to get container status \"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909\": rpc error: code = NotFound desc = could not find container \"232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909\": container with ID starting with 232100ab1a28f56332c98861c49f2d947519bff6d5b40ba9351a82cae0691909 not found: ID does not exist" Oct 30 21:42:03.394628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.394566 4921 scope.go:122] "RemoveContainer" containerID="bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c" Oct 30 21:42:03.408391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.408370 4921 scope.go:122] "RemoveContainer" containerID="bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c" Oct 30 21:42:03.408769 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:03.408739 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c\": container with ID starting with bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c not found: ID does not exist" containerID="bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c" Oct 30 21:42:03.408890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.408773 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c"} err="failed to get container status \"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c\": rpc error: code = NotFound desc = could not find container \"bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c\": container with ID starting with bdafa1dfa31c14bab80d8ef92f92031481c523bb19ec0b58c01421b06299610c not found: ID does not exist" Oct 30 21:42:03.408890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.408797 4921 scope.go:122] "RemoveContainer" containerID="201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623" Oct 30 21:42:03.431595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.431522 4921 scope.go:122] "RemoveContainer" containerID="201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623" Oct 30 21:42:03.431949 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:03.431927 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623\": container with ID starting with 201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623 not found: ID does not exist" containerID="201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623" Oct 30 21:42:03.432030 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.431956 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623"} err="failed to get container status \"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623\": rpc error: code = NotFound desc = could not find container \"201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623\": container with ID starting with 201b60a47f09638b80e9b0bdf40629d37b3132b6c289c390a36e10f27a67a623 not found: ID does not exist" Oct 30 21:42:03.432030 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.431973 4921 scope.go:122] "RemoveContainer" containerID="d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a" Oct 30 21:42:03.443909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.443882 4921 scope.go:122] "RemoveContainer" containerID="d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a" Oct 30 21:42:03.444189 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:03.444162 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a\": container with ID starting with d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a not found: ID does not exist" containerID="d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a" Oct 30 21:42:03.444255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.444201 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a"} err="failed to get container status \"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a\": rpc error: code = NotFound desc = could not find container \"d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a\": container with ID starting with d4f9050ce34b9849bdcbcf13d6a63831428642cc616670211357a8ee72e9b43a not found: ID does not exist" Oct 30 21:42:03.444255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.444222 4921 scope.go:122] "RemoveContainer" containerID="44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f" Oct 30 21:42:03.453576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.453552 4921 scope.go:122] "RemoveContainer" containerID="44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f" Oct 30 21:42:03.453810 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:03.453794 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f\": container with ID starting with 44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f not found: ID does not exist" containerID="44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f" Oct 30 21:42:03.453861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.453822 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f"} err="failed to get container status \"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f\": rpc error: code = NotFound desc = could not find container \"44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f\": container with ID starting with 44f16058f4011283cbe7844d7b4bf89c61904dc478dbe6c37d7794898f6b546f not found: ID does not exist" Oct 30 21:42:03.532062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.532035 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:42:03.536757 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.536726 4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:42:03.536864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.536785 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:42:03.537812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.537743 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-djgtl"] Oct 30 21:42:03.642505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.642436 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:42:03.646142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.646116 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-c5m7j"] Oct 30 21:42:03.754648 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.754620 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:42:03.758424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.758399 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-6t5kd"] Oct 30 21:42:03.865366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.865338 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:42:03.873305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.873283 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-svdn6"] Oct 30 21:42:03.981366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.981305 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:42:03.984821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:03.984800 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0a4eab-openshift-default-868987fdc7-9tzmm"] Oct 30 21:42:04.170753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:04.170719 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" path="/var/lib/kubelet/pods/006d8097-44b3-48d2-85f8-53ff8940234f/volumes" Oct 30 21:42:04.171256 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:04.171244 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" path="/var/lib/kubelet/pods/586eea25-2b06-4402-9c43-ebfe2bbe5caa/volumes" Oct 30 21:42:04.171713 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:04.171700 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" path="/var/lib/kubelet/pods/6bc2134f-56c5-49af-a0dc-97dc539d20e2/volumes" Oct 30 21:42:04.172650 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:04.172633 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" path="/var/lib/kubelet/pods/701aacc1-673e-4302-9bc3-a60d8fdd4405/volumes" Oct 30 21:42:04.173072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:04.173059 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" path="/var/lib/kubelet/pods/d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb/volumes" Oct 30 21:42:07.540572 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:07.540487 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:42:19.504158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.504125 4921 generic.go:356] "Generic (PLEG): container finished" podID="8d7bb186-b1d7-461f-a323-e3b945404994" containerID="e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7" exitCode=0 Oct 30 21:42:19.505018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.504198 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerDied","Data":"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7"} Oct 30 21:42:19.505018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.504567 4921 scope.go:122] "RemoveContainer" containerID="e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7" Oct 30 21:42:19.505973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.505941 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:19.506079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.506017 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:19.506079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:19.506067 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:20.531076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:20.531042 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerStarted","Data":"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad"} Oct 30 21:42:20.580670 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:20.580636 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:42:20.580989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:20.580962 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:42:20.581053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:20.581022 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 21:42:21.581125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:21.581091 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:42:21.581639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:21.581146 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 21:42:22.581203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:22.581172 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:42:22.581595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:22.581228 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 21:42:23.580905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:23.580870 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:42:23.581079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:23.580924 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 21:42:24.580998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:24.580967 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 21:42:24.581386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:24.581014 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 21:42:25.604247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:25.604213 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:42:25.606242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:25.606217 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:42:25.609957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:25.609934 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:42:37.541310 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:42:37.541266 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:42:43.166512 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:43.166474 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2"] Oct 30 21:42:43.215874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:43.215844 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6701073-jtxd2"] Oct 30 21:42:44.168536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:44.168504 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500c841f-479d-460c-8c3b-4417dcf00321" path="/var/lib/kubelet/pods/500c841f-479d-460c-8c3b-4417dcf00321/volumes" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.769250 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770233 4921 cpu_manager.go:414] "containerMap: removing container" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770256 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770286 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770295 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770311 4921 cpu_manager.go:414] "containerMap: removing container" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770480 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770500 4921 cpu_manager.go:414] "containerMap: removing container" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770509 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770545 4921 cpu_manager.go:414] "containerMap: removing container" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770554 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770583 4921 cpu_manager.go:414] "containerMap: removing container" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770590 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770638 4921 cpu_manager.go:414] "containerMap: removing container" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770647 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770896 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc2134f-56c5-49af-a0dc-97dc539d20e2" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770943 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2089674d-33e5-406a-8f53-55ece03fe7ef" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770959 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7369cab6-4366-42bb-8b8e-dfef7e7f9bc7" containerName="main" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770971 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="701aacc1-673e-4302-9bc3-a60d8fdd4405" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770981 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8143da2-8b43-40e9-a9a0-43e6cfdd8dbb" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.770997 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="006d8097-44b3-48d2-85f8-53ff8940234f" containerName="istio-proxy" Oct 30 21:42:46.773339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.771034 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="586eea25-2b06-4402-9c43-ebfe2bbe5caa" containerName="istio-proxy" Oct 30 21:42:46.796486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.796454 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.840469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.840415 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.841922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.841890 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.841969 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842031 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842073 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842110 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842203 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842181 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842653 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.842761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.842709 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z42q\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.857531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.857506 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e0f967-openshift-default-dockercfg-kjz7l\"" Oct 30 21:42:46.863397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.863366 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:42:46.944162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944132 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944195 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944247 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944307 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944356 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944401 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944470 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944531 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944575 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944589 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z42q\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.944861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944795 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.945018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.944997 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.945018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.945006 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.945156 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.945007 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.946864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.946838 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:46.947124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:46.947102 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:47.006611 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.006578 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z42q\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:47.018924 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.018890 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\") pod \"e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:47.121391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.121299 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:47.269164 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.269130 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:42:47.275272 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.275249 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.331527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.331494 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-99003c-openshift-default-dockercfg-vqvmj\"" Oct 30 21:42:47.337839 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.337817 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:42:47.368377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368347 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368390 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368412 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rr2\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368446 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368514 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368544 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368562 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368580 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.368875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.368692 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470071 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470323 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470136 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470323 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470169 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rr2\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470323 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470249 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470372 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470416 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470443 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470472 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470708 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470587 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470869 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.470979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.470955 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.471040 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.471019 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.471413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.471389 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.471526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.471462 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.477276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.477248 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.478075 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.478050 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.529788 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.529765 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.570290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.570254 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rr2\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\") pod \"e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.591462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.591434 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:47.616981 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:42:47.616893 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6973bd1c_bf6b_4653_8ee4_1eebe8234bf2.slice/crio-8f1e378b8bc19bf0cc038b07b8f2387c3822f2fd5c2b5cd114ccc7607413124e WatchSource:0}: Error finding container 8f1e378b8bc19bf0cc038b07b8f2387c3822f2fd5c2b5cd114ccc7607413124e: Status 404 returned error can't find the container with id 8f1e378b8bc19bf0cc038b07b8f2387c3822f2fd5c2b5cd114ccc7607413124e Oct 30 21:42:47.619975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.619933 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.620126 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.620022 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.620616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.620575 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:47.621901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.621871 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:42:47.837403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.837356 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerStarted","Data":"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e"} Oct 30 21:42:47.837871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.837422 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerStarted","Data":"8f1e378b8bc19bf0cc038b07b8f2387c3822f2fd5c2b5cd114ccc7607413124e"} Oct 30 21:42:47.977921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:47.977822 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podStartSLOduration=1.977804417 podStartE2EDuration="1.977804417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:47.975905875 +0000 UTC m=+1881.958606318" watchObservedRunningTime="2026-10-30 21:42:47.977804417 +0000 UTC m=+1881.960504835" Oct 30 21:42:48.044717 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:42:48.044683 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e48e41c_69b2_452e_af41_c50416fa9501.slice/crio-285093d9759bbd895251a854b9453db705f453ad7beba00df708f9de2e6f7b56 WatchSource:0}: Error finding container 285093d9759bbd895251a854b9453db705f453ad7beba00df708f9de2e6f7b56: Status 404 returned error can't find the container with id 285093d9759bbd895251a854b9453db705f453ad7beba00df708f9de2e6f7b56 Oct 30 21:42:48.045943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.045913 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:42:48.047404 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.047353 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.047526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.047428 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.047526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.047459 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:42:48.125632 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.124523 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:48.129160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.127843 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:42:48.129160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.127904 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:42:48.850930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.850895 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerStarted","Data":"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e"} Oct 30 21:42:48.851379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.850950 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerStarted","Data":"285093d9759bbd895251a854b9453db705f453ad7beba00df708f9de2e6f7b56"} Oct 30 21:42:48.999692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:48.999637 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podStartSLOduration=1.999620249 podStartE2EDuration="1.999620249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:48.999274984 +0000 UTC m=+1882.981975403" watchObservedRunningTime="2026-10-30 21:42:48.999620249 +0000 UTC m=+1882.982320667" Oct 30 21:42:49.133631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.133541 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:49.592363 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.592302 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:49.597607 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.597582 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:49.857017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.856934 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:49.857017 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.857006 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:42:49.857749 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.857725 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:42:49.857832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:42:49.857787 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:43:07.540818 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:43:07.540782 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:43:27.087232 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:27.087201 4921 scope.go:122] "RemoveContainer" containerID="2242983b4330b02bafc4bf80c7945e5a7dfa6c91d4abd90690d21bd2930872d5" Oct 30 21:43:37.551957 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:43:37.551869 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:43:46.160352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:46.160305 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:43:57.251797 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.251760 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:43:57.252339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.251988 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" containerID="cri-o://b2aec47519022a9eb592bc8565a2ed1b7ac27fe1cb8a1ff0d1e9a6017b474f5b" gracePeriod=30 Oct 30 21:43:57.263116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.263085 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:43:57.263278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.263259 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" containerID="cri-o://64e78a30ca860289ba808c7e9fa002d83b6a6a86c1b4e30c71d4691a6bec38e8" gracePeriod=30 Oct 30 21:43:57.284305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.284275 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:43:57.284470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:57.284454 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" containerID="cri-o://5dddfd2b00467acaad9f8eeb2ce7bf73bfd609b957a55db6eaff776fb5dbd98f" gracePeriod=30 Oct 30 21:43:58.470237 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:58.470196 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:43:58.470641 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:58.470259 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:43:58.950036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:58.950001 4921 patch_prober.go:29] interesting pod/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:43:58.950231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:43:58.950070 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:44:01.186892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.186854 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" containerID="cri-o://7b0e2da155eeeb417cd6a8abfd12774f4e294cda7d06c222740abeb1a19d9bea" gracePeriod=45 Oct 30 21:44:01.468756 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.468728 4921 generic.go:356] "Generic (PLEG): container finished" podID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerID="7b0e2da155eeeb417cd6a8abfd12774f4e294cda7d06c222740abeb1a19d9bea" exitCode=0 Oct 30 21:44:01.468912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.468772 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" event={"ID":"e9669ef5-a505-49fd-95e4-45a59fe4780e","Type":"ContainerDied","Data":"7b0e2da155eeeb417cd6a8abfd12774f4e294cda7d06c222740abeb1a19d9bea"} Oct 30 21:44:01.468912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.468797 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" event={"ID":"e9669ef5-a505-49fd-95e4-45a59fe4780e","Type":"ContainerDied","Data":"ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de"} Oct 30 21:44:01.468912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.468808 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7bc21a0788e8c7e93d2833d9f2d0c99090fb087990ed26c67188fcc2a6f9de" Oct 30 21:44:01.473110 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.473087 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:44:01.573427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573397 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573453 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573511 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\" (UniqueName: \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573560 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573637 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:01.573754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573713 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573770 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\") pod \"e9669ef5-a505-49fd-95e4-45a59fe4780e\" (UID: \"e9669ef5-a505-49fd-95e4-45a59fe4780e\") " Oct 30 21:44:01.573915 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.573878 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:01.574067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.574043 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:01.574262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.574243 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:01.574358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.574269 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:01.574358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.574282 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:01.575852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.575824 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:01.575852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.575839 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "kube-api-access-8cwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:01.575986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.575941 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs" pod "e9669ef5-a505-49fd-95e4-45a59fe4780e" (UID: "e9669ef5-a505-49fd-95e4-45a59fe4780e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:01.676595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.676558 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9669ef5-a505-49fd-95e4-45a59fe4780e-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:01.676595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.676593 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9669ef5-a505-49fd-95e4-45a59fe4780e-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:01.676843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:01.676607 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwp4\" (UniqueName: \"kubernetes.io/projected/e9669ef5-a505-49fd-95e4-45a59fe4780e-kube-api-access-8cwp4\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.483752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.483427 4921 generic.go:356] "Generic (PLEG): container finished" podID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerID="5dddfd2b00467acaad9f8eeb2ce7bf73bfd609b957a55db6eaff776fb5dbd98f" exitCode=0 Oct 30 21:44:02.483752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.483670 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" event={"ID":"48a2f12e-567e-4b4f-aeef-84b45c0668f3","Type":"ContainerDied","Data":"5dddfd2b00467acaad9f8eeb2ce7bf73bfd609b957a55db6eaff776fb5dbd98f"} Oct 30 21:44:02.485789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.485626 4921 generic.go:356] "Generic (PLEG): container finished" podID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerID="b2aec47519022a9eb592bc8565a2ed1b7ac27fe1cb8a1ff0d1e9a6017b474f5b" exitCode=0 Oct 30 21:44:02.485789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.485729 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" event={"ID":"7cd9dea5-a48f-404b-b888-809e9ac342b3","Type":"ContainerDied","Data":"b2aec47519022a9eb592bc8565a2ed1b7ac27fe1cb8a1ff0d1e9a6017b474f5b"} Oct 30 21:44:02.490486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.489425 4921 generic.go:356] "Generic (PLEG): container finished" podID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerID="64e78a30ca860289ba808c7e9fa002d83b6a6a86c1b4e30c71d4691a6bec38e8" exitCode=0 Oct 30 21:44:02.490486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.489506 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b" Oct 30 21:44:02.490486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.490074 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" event={"ID":"ade6ff81-281b-4e00-80d6-d6026fa257df","Type":"ContainerDied","Data":"64e78a30ca860289ba808c7e9fa002d83b6a6a86c1b4e30c71d4691a6bec38e8"} Oct 30 21:44:02.568138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.568109 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:44:02.589064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589036 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589091 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589133 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589182 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589246 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589275 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589512 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589517 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:02.589831 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589639 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.589831 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589796 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.589831 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589810 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.589929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589877 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\") pod \"ade6ff81-281b-4e00-80d6-d6026fa257df\" (UID: \"ade6ff81-281b-4e00-80d6-d6026fa257df\") " Oct 30 21:44:02.590021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.589998 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.591547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591367 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.591656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591640 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade6ff81-281b-4e00-80d6-d6026fa257df-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.591720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591668 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.591828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591804 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.591897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591848 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.591897 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.591865 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "kube-api-access-jvw2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.592780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.592756 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:02.597128 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.597078 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.604620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.604590 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token" pod "ade6ff81-281b-4e00-80d6-d6026fa257df" (UID: "ade6ff81-281b-4e00-80d6-d6026fa257df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.619198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.619177 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:44:02.636258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.636238 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:44:02.659507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.659487 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:44:02.692642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692620 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.692744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692657 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.692744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692682 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.692744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692731 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.692908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692771 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.692908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692817 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.692908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692879 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692925 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692961 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692986 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.693050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.692993 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:02.693050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693025 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693054 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.693293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693143 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.693293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693159 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.693293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693185 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693294 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693358 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693397 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693428 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693464 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\") pod \"7cd9dea5-a48f-404b-b888-809e9ac342b3\" (UID: \"7cd9dea5-a48f-404b-b888-809e9ac342b3\") " Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693526 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693556 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\") pod \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\" (UID: \"48a2f12e-567e-4b4f-aeef-84b45c0668f3\") " Oct 30 21:44:02.693928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.693644 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694122 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694148 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694155 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694163 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694220 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694238 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694245 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694253 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694283 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694380 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694396 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw2h\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-kube-api-access-jvw2h\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694401 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694413 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694444 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade6ff81-281b-4e00-80d6-d6026fa257df-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.694702 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.694288 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:02.695612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.695387 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.695805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.695782 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.695889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.695866 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:02.696241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.696219 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:02.696519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.696488 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:02.696693 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.696676 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8" pod "48a2f12e-567e-4b4f-aeef-84b45c0668f3" (UID: "48a2f12e-567e-4b4f-aeef-84b45c0668f3"). InnerVolumeSpecName "kube-api-access-q8ks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.696873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.696857 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "kube-api-access-4hz8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.697174 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.697156 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token" pod "7cd9dea5-a48f-404b-b888-809e9ac342b3" (UID: "7cd9dea5-a48f-404b-b888-809e9ac342b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:02.708305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.708251 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0a4eab/simulator-w5-0a4eab-kserve-75d65bd5d5-4tg2b"] Oct 30 21:44:02.795944 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.795615 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hz8z\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-kube-api-access-4hz8z\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796095 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ks8\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-kube-api-access-q8ks8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796119 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796133 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796147 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796160 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796173 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796185 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd9dea5-a48f-404b-b888-809e9ac342b3-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796199 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796199 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796212 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796226 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a2f12e-567e-4b4f-aeef-84b45c0668f3-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:02.796505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:02.796241 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd9dea5-a48f-404b-b888-809e9ac342b3-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:03.058118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.058040 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:44:03.159353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.159300 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:44:03.528022 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.527993 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" Oct 30 21:44:03.528500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.528024 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6" event={"ID":"7cd9dea5-a48f-404b-b888-809e9ac342b3","Type":"ContainerDied","Data":"bb45dcad6a3b91cb28ebede5ee0eb1ab9ce4da766129e95710a4642501d37527"} Oct 30 21:44:03.528500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.528085 4921 scope.go:122] "RemoveContainer" containerID="b2aec47519022a9eb592bc8565a2ed1b7ac27fe1cb8a1ff0d1e9a6017b474f5b" Oct 30 21:44:03.529986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.529961 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" Oct 30 21:44:03.530109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.529961 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr" event={"ID":"ade6ff81-281b-4e00-80d6-d6026fa257df","Type":"ContainerDied","Data":"1b25bfcb1dc6ff1ee8ec1467d2a8eb618d67e51d8acf4d9983bf1107c6130599"} Oct 30 21:44:03.531829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.531802 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" event={"ID":"48a2f12e-567e-4b4f-aeef-84b45c0668f3","Type":"ContainerDied","Data":"3f764e02e1e50fed74b8909753fbe82c7e0b9bfabf75b3caf1b83cf61201f81a"} Oct 30 21:44:03.531829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.531826 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2" Oct 30 21:44:03.542234 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.542196 4921 scope.go:122] "RemoveContainer" containerID="64e78a30ca860289ba808c7e9fa002d83b6a6a86c1b4e30c71d4691a6bec38e8" Oct 30 21:44:03.555184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.555161 4921 scope.go:122] "RemoveContainer" containerID="5dddfd2b00467acaad9f8eeb2ce7bf73bfd609b957a55db6eaff776fb5dbd98f" Oct 30 21:44:03.708364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.708328 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:44:03.728721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.728692 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-9g9q2"] Oct 30 21:44:03.841180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.841110 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:44:03.846502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.846480 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-sxnlr"] Oct 30 21:44:03.955665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.955630 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:44:03.959025 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:03.959001 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9ac5c5-openshift-default-5f9f955bc7-r8xh6"] Oct 30 21:44:04.170079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:04.170005 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" path="/var/lib/kubelet/pods/48a2f12e-567e-4b4f-aeef-84b45c0668f3/volumes" Oct 30 21:44:04.170565 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:04.170550 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" path="/var/lib/kubelet/pods/7cd9dea5-a48f-404b-b888-809e9ac342b3/volumes" Oct 30 21:44:04.171024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:04.171010 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" path="/var/lib/kubelet/pods/ade6ff81-281b-4e00-80d6-d6026fa257df/volumes" Oct 30 21:44:04.171752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:04.171738 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" path="/var/lib/kubelet/pods/e9669ef5-a505-49fd-95e4-45a59fe4780e/volumes" Oct 30 21:44:07.541037 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:07.541002 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:44:08.058424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:08.058386 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" containerID="cri-o://3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4" gracePeriod=25 Oct 30 21:44:08.159921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:08.159883 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" containerID="cri-o://ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798" gracePeriod=25 Oct 30 21:44:13.861204 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:13.861163 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:44:13.861643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:13.861217 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:44:20.900462 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:20.900423 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:44:20.900963 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:20.900492 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:22.830717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.830683 4921 generic.go:356] "Generic (PLEG): container finished" podID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerID="9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e" exitCode=0 Oct 30 21:44:22.831152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.830754 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerDied","Data":"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e"} Oct 30 21:44:22.831152 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.831084 4921 scope.go:122] "RemoveContainer" containerID="9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e" Oct 30 21:44:22.832532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.832498 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:22.832655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.832607 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:22.832722 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:22.832663 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:23.840307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:23.840277 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerStarted","Data":"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa"} Oct 30 21:44:23.860789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:23.860760 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:44:23.860983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:23.860824 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:44:24.128397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:24.127537 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:44:24.128397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:24.127802 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:44:24.128397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:24.127868 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:44:25.123671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:25.123636 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:44:25.124089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:25.123696 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:44:26.124147 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:26.124097 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:44:26.124716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:26.124152 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:44:27.123338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.122463 4921 scope.go:122] "RemoveContainer" containerID="8a6c1ed45a3d574fb6086d11ae557b8c4c16584f358c35f35040e8a18c969563" Oct 30 21:44:27.123338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.122822 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:44:27.123338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.122878 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:44:27.136957 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.136933 4921 scope.go:122] "RemoveContainer" containerID="d40c195f43fa03b6b897c9d7b76a49353f228929c03805a1e67a534429c8bffe" Oct 30 21:44:27.146435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.146416 4921 scope.go:122] "RemoveContainer" containerID="7b0e2da155eeeb417cd6a8abfd12774f4e294cda7d06c222740abeb1a19d9bea" Oct 30 21:44:27.155413 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:27.155394 4921 scope.go:122] "RemoveContainer" containerID="6f56ed499d5d5467287ddecf868d23ecd66d1692c2f43d3976747a74abc189ee" Oct 30 21:44:28.124073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.123752 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:44:28.124073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.123794 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:44:28.392200 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.392171 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:44:28.486168 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.486146 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:44:28.490090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.490072 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\") pod \"e13f9ac9-43d1-428a-b248-0d513323f20b\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " Oct 30 21:44:28.490542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.490525 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\" (UniqueName: \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\") pod \"e13f9ac9-43d1-428a-b248-0d513323f20b\" (UID: \"e13f9ac9-43d1-428a-b248-0d513323f20b\") " Oct 30 21:44:28.491099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.490477 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume" pod "e13f9ac9-43d1-428a-b248-0d513323f20b" (UID: "e13f9ac9-43d1-428a-b248-0d513323f20b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:28.491655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.491638 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13f9ac9-43d1-428a-b248-0d513323f20b-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:28.493077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.493050 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r" pod "e13f9ac9-43d1-428a-b248-0d513323f20b" (UID: "e13f9ac9-43d1-428a-b248-0d513323f20b"). InnerVolumeSpecName "kube-api-access-mng6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:28.592404 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.592376 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\" (UniqueName: \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\") pod \"2115dda4-e1d6-47ad-a064-e961685640b1\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " Oct 30 21:44:28.592588 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.592459 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\") pod \"2115dda4-e1d6-47ad-a064-e961685640b1\" (UID: \"2115dda4-e1d6-47ad-a064-e961685640b1\") " Oct 30 21:44:28.592807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.592786 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mng6r\" (UniqueName: \"kubernetes.io/projected/e13f9ac9-43d1-428a-b248-0d513323f20b-kube-api-access-mng6r\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:28.593133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.593114 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume" pod "2115dda4-e1d6-47ad-a064-e961685640b1" (UID: "2115dda4-e1d6-47ad-a064-e961685640b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:28.594800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.594778 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd" pod "2115dda4-e1d6-47ad-a064-e961685640b1" (UID: "2115dda4-e1d6-47ad-a064-e961685640b1"). InnerVolumeSpecName "kube-api-access-rj2zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:28.694552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.694479 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2zd\" (UniqueName: \"kubernetes.io/projected/2115dda4-e1d6-47ad-a064-e961685640b1-kube-api-access-rj2zd\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:28.694552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.694512 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2115dda4-e1d6-47ad-a064-e961685640b1-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:28.880038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.880006 4921 generic.go:356] "Generic (PLEG): container finished" podID="2115dda4-e1d6-47ad-a064-e961685640b1" containerID="ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798" exitCode=0 Oct 30 21:44:28.880230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.880051 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" Oct 30 21:44:28.880230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.880089 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" event={"ID":"2115dda4-e1d6-47ad-a064-e961685640b1","Type":"ContainerDied","Data":"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798"} Oct 30 21:44:28.880230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.880143 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8" event={"ID":"2115dda4-e1d6-47ad-a064-e961685640b1","Type":"ContainerDied","Data":"44afe8ef80cfe52cf4b046213887bdf343bc58fd6e07f3f2a9db006b9dd12ae9"} Oct 30 21:44:28.880230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.880172 4921 scope.go:122] "RemoveContainer" containerID="ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798" Oct 30 21:44:28.881698 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.881678 4921 generic.go:356] "Generic (PLEG): container finished" podID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerID="3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4" exitCode=0 Oct 30 21:44:28.881802 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.881784 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" Oct 30 21:44:28.881860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.881792 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" event={"ID":"e13f9ac9-43d1-428a-b248-0d513323f20b","Type":"ContainerDied","Data":"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4"} Oct 30 21:44:28.881918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.881880 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt" event={"ID":"e13f9ac9-43d1-428a-b248-0d513323f20b","Type":"ContainerDied","Data":"e5b65d0887fa65e386a094ed653f3098b6f7102db9f2ace65ac5a130747b9046"} Oct 30 21:44:28.892765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.891448 4921 scope.go:122] "RemoveContainer" containerID="ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798" Oct 30 21:44:28.893156 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:28.893116 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798\": container with ID starting with ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798 not found: ID does not exist" containerID="ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798" Oct 30 21:44:28.893267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.893158 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798"} err="failed to get container status \"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798\": rpc error: code = NotFound desc = could not find container \"ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798\": container with ID starting with ab0e7edfc48ace0939e280d5b95ff66b99249b6072e47fe378a7fb9a7b36d798 not found: ID does not exist" Oct 30 21:44:28.893267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.893182 4921 scope.go:122] "RemoveContainer" containerID="3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4" Oct 30 21:44:28.907223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.907201 4921 scope.go:122] "RemoveContainer" containerID="3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4" Oct 30 21:44:28.907534 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:28.907514 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4\": container with ID starting with 3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4 not found: ID does not exist" containerID="3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4" Oct 30 21:44:28.907590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:28.907543 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4"} err="failed to get container status \"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4\": rpc error: code = NotFound desc = could not find container \"3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4\": container with ID starting with 3511cb0bde59eb8ca56259d55bb5e064e926c22da486fe3a0e32f354f544d6e4 not found: ID does not exist" Oct 30 21:44:29.049158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.049127 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:44:29.054811 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.054782 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-0a4eab-5996db6ddc-n8sgt"] Oct 30 21:44:29.126343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.126304 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:44:29.126743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.126715 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:44:29.127494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.127478 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:44:29.159728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.159705 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:44:29.163266 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:29.163242 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-0a4eab-f897ff5c5-69lw8"] Oct 30 21:44:30.169183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:30.169154 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" path="/var/lib/kubelet/pods/2115dda4-e1d6-47ad-a064-e961685640b1/volumes" Oct 30 21:44:30.169597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:30.169580 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" path="/var/lib/kubelet/pods/e13f9ac9-43d1-428a-b248-0d513323f20b/volumes" Oct 30 21:44:33.488073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488035 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:44:33.488830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488806 4921 cpu_manager.go:414] "containerMap: removing container" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" Oct 30 21:44:33.488830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488829 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488852 4921 cpu_manager.go:414] "containerMap: removing container" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488859 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488872 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488879 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488891 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488897 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488941 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488947 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488980 4921 cpu_manager.go:414] "containerMap: removing container" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" Oct 30 21:44:33.488993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.488988 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489137 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2115dda4-e1d6-47ad-a064-e961685640b1" containerName="payload-pre-processing" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489152 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade6ff81-281b-4e00-80d6-d6026fa257df" containerName="istio-proxy" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489166 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a2f12e-567e-4b4f-aeef-84b45c0668f3" containerName="istio-proxy" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489176 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd9dea5-a48f-404b-b888-809e9ac342b3" containerName="istio-proxy" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489183 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13f9ac9-43d1-428a-b248-0d513323f20b" containerName="payload-processing" Oct 30 21:44:33.489479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.489191 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9669ef5-a505-49fd-95e4-45a59fe4780e" containerName="main" Oct 30 21:44:33.492986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.492965 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.544804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.544770 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-fe5385-serving-cert\"" Oct 30 21:44:33.554930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.554902 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.555045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.554950 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.555045 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.555019 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7m75\" (UniqueName: \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.555188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.555168 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.597777 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.597753 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:44:33.656197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.656168 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.656365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.656309 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.656365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.656350 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.656497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.656371 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7m75\" (UniqueName: \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.656497 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:33.656440 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-serving-cert: secret "maas-api-e2e-shared-a-w0-fe5385-serving-cert" not found Oct 30 21:44:33.656607 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:33.656524 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls podName:0beb01b5-0304-47d4-b704-e6510860a6b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:34.156504511 +0000 UTC m=+1988.139204932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls") pod "maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" (UID: "0beb01b5-0304-47d4-b704-e6510860a6b8") : secret "maas-api-e2e-shared-a-w0-fe5385-serving-cert" not found Oct 30 21:44:33.657023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.657001 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.658577 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.658558 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:33.723337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:33.714618 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7m75\" (UniqueName: \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:34.163889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:34.163854 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:34.166821 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:34.166790 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:34.407744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:34.407704 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:34.790029 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:34.789995 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:44:34.790483 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:44:34.790437 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0beb01b5_0304_47d4_b704_e6510860a6b8.slice/crio-1758ee1e2994a71246d0525266fec2c757200ce184664ee445c2ffb86d59f768 WatchSource:0}: Error finding container 1758ee1e2994a71246d0525266fec2c757200ce184664ee445c2ffb86d59f768: Status 404 returned error can't find the container with id 1758ee1e2994a71246d0525266fec2c757200ce184664ee445c2ffb86d59f768 Oct 30 21:44:34.931860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:34.931820 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" event={"ID":"0beb01b5-0304-47d4-b704-e6510860a6b8","Type":"ContainerStarted","Data":"1758ee1e2994a71246d0525266fec2c757200ce184664ee445c2ffb86d59f768"} Oct 30 21:44:35.941207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:35.941111 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" event={"ID":"0beb01b5-0304-47d4-b704-e6510860a6b8","Type":"ContainerStarted","Data":"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde"} Oct 30 21:44:35.941625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:35.941201 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:36.062981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:36.062926 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" podStartSLOduration=2.240735853 podStartE2EDuration="3.06291281s" totalImagesPullingTime="822.176957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:36.059859873 +0000 UTC m=+1990.042560292" watchObservedRunningTime="2026-10-30 21:44:36.06291281 +0000 UTC m=+1990.045613227" Oct 30 21:44:37.540902 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:37.540860 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:44:41.956242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:41.956210 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:44:48.199332 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:48.199287 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:44:48.200602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:48.200568 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" containerID="cri-o://b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06" gracePeriod=30 Oct 30 21:44:49.096300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.096204 4921 generic.go:356] "Generic (PLEG): container finished" podID="8e48e41c-69b2-452e-af41-c50416fa9501" containerID="8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e" exitCode=0 Oct 30 21:44:49.096300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.096282 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerDied","Data":"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e"} Oct 30 21:44:49.096743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.096727 4921 scope.go:122] "RemoveContainer" containerID="8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e" Oct 30 21:44:49.098171 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.098137 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:49.098276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.098205 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:49.098276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.098246 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:44:49.856790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:49.856745 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:44:50.132776 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:50.132679 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerStarted","Data":"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407"} Oct 30 21:44:50.592036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:50.592002 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:44:50.593490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:50.593462 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:44:50.593649 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:50.593517 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:44:51.592816 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:51.592783 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:44:51.593195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:51.592844 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:44:52.591938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:52.591904 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:44:52.592148 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:52.591963 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:44:53.533510 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.533421 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:44:53.569934 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.569897 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.570281 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.570256 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:53.570397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.570367 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.570850 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.570828 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571171 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571233 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571260 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571334 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571389 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.571537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.571482 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\") pod \"78820725-24be-428c-a0e0-1a4ccf83f479\" (UID: \"78820725-24be-428c-a0e0-1a4ccf83f479\") " Oct 30 21:44:53.572384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572196 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78820725-24be-428c-a0e0-1a4ccf83f479-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.572384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572269 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:53.572384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572220 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:53.572633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572410 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:53.572633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572591 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:53.572958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.572935 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "kube-api-access-4tnw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:53.574114 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.574088 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:53.576253 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.576225 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:44:53.576405 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.576381 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token" pod "78820725-24be-428c-a0e0-1a4ccf83f479" (UID: "78820725-24be-428c-a0e0-1a4ccf83f479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:53.592260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.591961 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:44:53.592260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.592011 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:44:53.673306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673275 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnw6\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-kube-api-access-4tnw6\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673302 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673324 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673336 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78820725-24be-428c-a0e0-1a4ccf83f479-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673346 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78820725-24be-428c-a0e0-1a4ccf83f479-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673354 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673361 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.673538 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:53.673369 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78820725-24be-428c-a0e0-1a4ccf83f479-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:54.168364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.167901 4921 generic.go:356] "Generic (PLEG): container finished" podID="78820725-24be-428c-a0e0-1a4ccf83f479" containerID="b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06" exitCode=0 Oct 30 21:44:54.168364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.167989 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" Oct 30 21:44:54.170434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.170411 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" event={"ID":"78820725-24be-428c-a0e0-1a4ccf83f479","Type":"ContainerDied","Data":"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06"} Oct 30 21:44:54.170559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.170450 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2" event={"ID":"78820725-24be-428c-a0e0-1a4ccf83f479","Type":"ContainerDied","Data":"c6ecde31b7ae0723bdc4aaffdedd5e96d33e13318c1ce54555c1f960bf267d60"} Oct 30 21:44:54.170559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.170472 4921 scope.go:122] "RemoveContainer" containerID="b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06" Oct 30 21:44:54.182694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.182677 4921 scope.go:122] "RemoveContainer" containerID="b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06" Oct 30 21:44:54.182979 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:44:54.182958 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06\": container with ID starting with b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06 not found: ID does not exist" containerID="b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06" Oct 30 21:44:54.183050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.183007 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06"} err="failed to get container status \"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06\": rpc error: code = NotFound desc = could not find container \"b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06\": container with ID starting with b68781d942ad90fee7516089532655342b0606783369df6cece16a42800edc06 not found: ID does not exist" Oct 30 21:44:54.335187 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.335157 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:44:54.338850 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.338822 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-847c62-openshift-default-6bc5bf44df-zq9n2"] Oct 30 21:44:54.592491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.592462 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.592905 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:54.592515 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 21:44:55.596681 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:55.596654 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:44:55.597184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:55.597155 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:44:55.597871 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:55.597852 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:44:56.169458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:44:56.169428 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" path="/var/lib/kubelet/pods/78820725-24be-428c-a0e0-1a4ccf83f479/volumes" Oct 30 21:45:00.182089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.182055 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:00.185676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.183015 4921 cpu_manager.go:414] "containerMap: removing container" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" Oct 30 21:45:00.185676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.183034 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" Oct 30 21:45:00.185676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.183227 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="78820725-24be-428c-a0e0-1a4ccf83f479" containerName="istio-proxy" Oct 30 21:45:00.189305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.189256 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.191477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.191450 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:00.198616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.198590 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.205958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.205932 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:00.215403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.214255 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.261254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261221 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.261399 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261254 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9l5x\" (UniqueName: \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.261464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261444 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72h5\" (UniqueName: \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.261636 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261610 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.261695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261678 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fqp\" (UniqueName: \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.261848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.261827 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.271497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.271474 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:00.326461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.326433 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:00.330862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.330839 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:00.351220 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.351193 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:45:00.363077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363039 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72h5\" (UniqueName: \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.363201 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363159 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.363267 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363207 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fqp\" (UniqueName: \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.363343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363285 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.363485 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363463 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.363548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.363503 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9l5x\" (UniqueName: \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.364085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.364051 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.364194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.364169 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.364194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.364175 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.419438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.419408 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fqp\" (UniqueName: \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.419653 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.419634 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72h5\" (UniqueName: \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\") pod \"maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.423956 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.423935 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9l5x\" (UniqueName: \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.524087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.523928 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:00.534226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.534206 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:00.540372 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.540352 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:00.911330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.911291 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:00.911445 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:45:00.911398 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9342e918_b556_4b4b_83b1_b96a704566a2.slice/crio-453aeca64af8590895d72998c0abeca433555fb570ee0bf8f619436d862f2fa1 WatchSource:0}: Error finding container 453aeca64af8590895d72998c0abeca433555fb570ee0bf8f619436d862f2fa1: Status 404 returned error can't find the container with id 453aeca64af8590895d72998c0abeca433555fb570ee0bf8f619436d862f2fa1 Oct 30 21:45:00.961586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.961462 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:00.962132 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:45:00.962104 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9313f73_9a4e_4b98_a376_4fa2bb9ae7d1.slice/crio-bfd0eab7ef911b72a433656e9f11530d61dc3033122a1211d31f72f6d73ccade WatchSource:0}: Error finding container bfd0eab7ef911b72a433656e9f11530d61dc3033122a1211d31f72f6d73ccade: Status 404 returned error can't find the container with id bfd0eab7ef911b72a433656e9f11530d61dc3033122a1211d31f72f6d73ccade Oct 30 21:45:00.963719 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:45:00.963697 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f1ade2_11af_4121_822b_9067b8740ce4.slice/crio-19c9ecc8139794ffe557139cea2e7eff6c25c03973cef248214c9d9bc202c943 WatchSource:0}: Error finding container 19c9ecc8139794ffe557139cea2e7eff6c25c03973cef248214c9d9bc202c943: Status 404 returned error can't find the container with id 19c9ecc8139794ffe557139cea2e7eff6c25c03973cef248214c9d9bc202c943 Oct 30 21:45:00.965528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:00.965508 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:01.239127 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:01.239098 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerStarted","Data":"19c9ecc8139794ffe557139cea2e7eff6c25c03973cef248214c9d9bc202c943"} Oct 30 21:45:01.240901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:01.240855 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerStarted","Data":"669d468afd20b333d12d5df831ad22ed54a9cc094474700e8d61025115b11577"} Oct 30 21:45:01.240999 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:01.240928 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerStarted","Data":"453aeca64af8590895d72998c0abeca433555fb570ee0bf8f619436d862f2fa1"} Oct 30 21:45:01.242155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:01.242133 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" event={"ID":"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1","Type":"ContainerStarted","Data":"bfd0eab7ef911b72a433656e9f11530d61dc3033122a1211d31f72f6d73ccade"} Oct 30 21:45:01.353604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:01.353559 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" podStartSLOduration=1.353536127 podStartE2EDuration="1.353536127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.353139048 +0000 UTC m=+2015.335839479" watchObservedRunningTime="2026-10-30 21:45:01.353536127 +0000 UTC m=+2015.336236546" Oct 30 21:45:02.249604 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.249576 4921 generic.go:356] "Generic (PLEG): container finished" podID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerID="20d5e3de0ac97d79cf754041b669c8bf4b8adc787a900748c7a6d42394fae90a" exitCode=60 Oct 30 21:45:02.249995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.249659 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" event={"ID":"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1","Type":"ContainerDied","Data":"20d5e3de0ac97d79cf754041b669c8bf4b8adc787a900748c7a6d42394fae90a"} Oct 30 21:45:02.249995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.249859 4921 scope.go:122] "RemoveContainer" containerID="20d5e3de0ac97d79cf754041b669c8bf4b8adc787a900748c7a6d42394fae90a" Oct 30 21:45:02.251501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.251479 4921 generic.go:356] "Generic (PLEG): container finished" podID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerID="f729f429dd335d7e9e5cb91ac1c4b076733e2062057a2984e5827c0f9d429b49" exitCode=60 Oct 30 21:45:02.251593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.251564 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerDied","Data":"f729f429dd335d7e9e5cb91ac1c4b076733e2062057a2984e5827c0f9d429b49"} Oct 30 21:45:02.251752 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.251732 4921 scope.go:122] "RemoveContainer" containerID="f729f429dd335d7e9e5cb91ac1c4b076733e2062057a2984e5827c0f9d429b49" Oct 30 21:45:02.253290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.253268 4921 generic.go:356] "Generic (PLEG): container finished" podID="9342e918-b556-4b4b-83b1-b96a704566a2" containerID="669d468afd20b333d12d5df831ad22ed54a9cc094474700e8d61025115b11577" exitCode=60 Oct 30 21:45:02.253442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.253415 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerDied","Data":"669d468afd20b333d12d5df831ad22ed54a9cc094474700e8d61025115b11577"} Oct 30 21:45:02.253593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:02.253577 4921 scope.go:122] "RemoveContainer" containerID="669d468afd20b333d12d5df831ad22ed54a9cc094474700e8d61025115b11577" Oct 30 21:45:03.262496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.262463 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerStarted","Data":"4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5"} Oct 30 21:45:03.264377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.264354 4921 generic.go:356] "Generic (PLEG): container finished" podID="9342e918-b556-4b4b-83b1-b96a704566a2" containerID="3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808" exitCode=60 Oct 30 21:45:03.264484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.264428 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerDied","Data":"3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808"} Oct 30 21:45:03.264536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.264481 4921 scope.go:122] "RemoveContainer" containerID="669d468afd20b333d12d5df831ad22ed54a9cc094474700e8d61025115b11577" Oct 30 21:45:03.264633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.264605 4921 scope.go:122] "RemoveContainer" containerID="3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808" Oct 30 21:45:03.265086 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:03.264911 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng_odh-ai-gateway-infra(9342e918-b556-4b4b-83b1-b96a704566a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" Oct 30 21:45:03.266540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.266520 4921 generic.go:356] "Generic (PLEG): container finished" podID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerID="c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7" exitCode=60 Oct 30 21:45:03.266626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.266607 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" event={"ID":"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1","Type":"ContainerDied","Data":"c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7"} Oct 30 21:45:03.266772 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.266758 4921 scope.go:122] "RemoveContainer" containerID="c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7" Oct 30 21:45:03.266966 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:03.266949 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6_odh-ai-gateway-infra(f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" Oct 30 21:45:03.277071 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.277054 4921 scope.go:122] "RemoveContainer" containerID="20d5e3de0ac97d79cf754041b669c8bf4b8adc787a900748c7a6d42394fae90a" Oct 30 21:45:03.380449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:03.380372 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" podStartSLOduration=3.380357476 podStartE2EDuration="3.380357476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.375925709 +0000 UTC m=+2017.358626125" watchObservedRunningTime="2026-10-30 21:45:03.380357476 +0000 UTC m=+2017.363057894" Oct 30 21:45:04.276061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.276028 4921 generic.go:356] "Generic (PLEG): container finished" podID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerID="4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5" exitCode=60 Oct 30 21:45:04.276542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.276101 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerDied","Data":"4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5"} Oct 30 21:45:04.276542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.276150 4921 scope.go:122] "RemoveContainer" containerID="f729f429dd335d7e9e5cb91ac1c4b076733e2062057a2984e5827c0f9d429b49" Oct 30 21:45:04.276542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.276293 4921 scope.go:122] "RemoveContainer" containerID="4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5" Oct 30 21:45:04.276713 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:04.276600 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx_odh-ai-gateway-infra(e7f1ade2-11af-4121-822b-9067b8740ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" Oct 30 21:45:04.278039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.278018 4921 scope.go:122] "RemoveContainer" containerID="3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808" Oct 30 21:45:04.278269 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:04.278248 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng_odh-ai-gateway-infra(9342e918-b556-4b4b-83b1-b96a704566a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" Oct 30 21:45:04.279652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:04.279633 4921 scope.go:122] "RemoveContainer" containerID="c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7" Oct 30 21:45:04.279840 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:04.279824 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6_odh-ai-gateway-infra(f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" Oct 30 21:45:05.288930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:05.288905 4921 scope.go:122] "RemoveContainer" containerID="4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5" Oct 30 21:45:05.289349 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:05.289158 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx_odh-ai-gateway-infra(e7f1ade2-11af-4121-822b-9067b8740ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" Oct 30 21:45:07.540302 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:07.540218 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:45:14.161716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:14.161686 4921 scope.go:122] "RemoveContainer" containerID="3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808" Oct 30 21:45:15.359747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:15.359674 4921 generic.go:356] "Generic (PLEG): container finished" podID="9342e918-b556-4b4b-83b1-b96a704566a2" containerID="edaa97bebcccce7553d1ef81d1638a5cd3511f2623b2699a3f67de6a2b7da274" exitCode=60 Oct 30 21:45:15.360144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:15.359747 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerDied","Data":"edaa97bebcccce7553d1ef81d1638a5cd3511f2623b2699a3f67de6a2b7da274"} Oct 30 21:45:15.360144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:15.359829 4921 scope.go:122] "RemoveContainer" containerID="3549766a5d5513da7001b5ae4ea0ce71603d621d00c0ecd7424b36e5097c1808" Oct 30 21:45:15.360293 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:15.360175 4921 scope.go:122] "RemoveContainer" containerID="edaa97bebcccce7553d1ef81d1638a5cd3511f2623b2699a3f67de6a2b7da274" Oct 30 21:45:15.360524 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:15.360500 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng_odh-ai-gateway-infra(9342e918-b556-4b4b-83b1-b96a704566a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" Oct 30 21:45:16.165431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.165400 4921 scope.go:122] "RemoveContainer" containerID="c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7" Oct 30 21:45:16.489099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.489060 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:16.653994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.653966 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:16.769368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.769291 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\") pod \"9342e918-b556-4b4b-83b1-b96a704566a2\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " Oct 30 21:45:16.769695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.769486 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\" (UniqueName: \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\") pod \"9342e918-b556-4b4b-83b1-b96a704566a2\" (UID: \"9342e918-b556-4b4b-83b1-b96a704566a2\") " Oct 30 21:45:16.770669 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.770625 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca" pod "9342e918-b556-4b4b-83b1-b96a704566a2" (UID: "9342e918-b556-4b4b-83b1-b96a704566a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:16.775819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.775791 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp" pod "9342e918-b556-4b4b-83b1-b96a704566a2" (UID: "9342e918-b556-4b4b-83b1-b96a704566a2"). InnerVolumeSpecName "kube-api-access-w8fqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:16.871427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.871402 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9342e918-b556-4b4b-83b1-b96a704566a2-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:16.871427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:16.871424 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fqp\" (UniqueName: \"kubernetes.io/projected/9342e918-b556-4b4b-83b1-b96a704566a2-kube-api-access-w8fqp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.069213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.069144 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:45:17.069862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.069845 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.069912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.069865 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.069912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.069883 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.069912 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.069888 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.070020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.070009 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.070053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.070026 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:17.073420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.073401 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.129738 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.129666 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3ab88248-openshift-default-dockercfg-djf4d\"" Oct 30 21:45:17.176247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.176012 4921 scope.go:122] "RemoveContainer" containerID="4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186306 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186391 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186420 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186446 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh5p\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186506 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186535 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186587 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186755 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.187024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.186806 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.206824 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.206541 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:45:17.288151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288116 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288202 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288302 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288356 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288377 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288394 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh5p\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288411 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288430 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288590 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288467 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.288946 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.288922 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.289212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.289191 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.289813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.289790 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.290790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.290769 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.291057 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.291034 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.292558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.292535 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.292864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.292839 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.348760 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.348723 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.358215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.358184 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh5p\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\") pod \"e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.377786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.377764 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" Oct 30 21:45:17.377891 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.377801 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng" event={"ID":"9342e918-b556-4b4b-83b1-b96a704566a2","Type":"ContainerDied","Data":"453aeca64af8590895d72998c0abeca433555fb570ee0bf8f619436d862f2fa1"} Oct 30 21:45:17.377891 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.377865 4921 scope.go:122] "RemoveContainer" containerID="edaa97bebcccce7553d1ef81d1638a5cd3511f2623b2699a3f67de6a2b7da274" Oct 30 21:45:17.379993 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.379972 4921 generic.go:356] "Generic (PLEG): container finished" podID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerID="177b3188dcfa6ff294d80a9cf0a51e987ad634cce7e3b9184f00b2e35e396f22" exitCode=60 Oct 30 21:45:17.380095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.380052 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" event={"ID":"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1","Type":"ContainerDied","Data":"177b3188dcfa6ff294d80a9cf0a51e987ad634cce7e3b9184f00b2e35e396f22"} Oct 30 21:45:17.380428 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.380414 4921 scope.go:122] "RemoveContainer" containerID="177b3188dcfa6ff294d80a9cf0a51e987ad634cce7e3b9184f00b2e35e396f22" Oct 30 21:45:17.380694 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:17.380676 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6_odh-ai-gateway-infra(f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" Oct 30 21:45:17.388820 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.388799 4921 scope.go:122] "RemoveContainer" containerID="c8a779e56e2ef871cdac4cf58b6e142c6cd3e8e3ef3dcb20cc99409f792c52a7" Oct 30 21:45:17.392347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.389950 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:17.542725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.542354 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:17.553426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.553400 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7cf226-29889945-zkkng"] Oct 30 21:45:17.926864 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.926824 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:45:17.927072 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:45:17.927050 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b07a77a_abde_4656_a9cb_539187d89e50.slice/crio-3e5c72ea10c7db14437516c2ab842f57dca944dc6eb5cd32b8a6972bbdf50d25 WatchSource:0}: Error finding container 3e5c72ea10c7db14437516c2ab842f57dca944dc6eb5cd32b8a6972bbdf50d25: Status 404 returned error can't find the container with id 3e5c72ea10c7db14437516c2ab842f57dca944dc6eb5cd32b8a6972bbdf50d25 Oct 30 21:45:17.930050 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.930012 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:17.930191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.930076 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:17.930191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:17.930109 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:18.170534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.170502 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" path="/var/lib/kubelet/pods/9342e918-b556-4b4b-83b1-b96a704566a2/volumes" Oct 30 21:45:18.391018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.390970 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerStarted","Data":"d99e5bbcbf2370ec833252f754b7c82d7e547bf5bd478e5cfb04e49782fbe936"} Oct 30 21:45:18.394144 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.394115 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerStarted","Data":"f411ff91e86e6845834c50379174f3b64b7504fc2511145d7dfa7913af0354da"} Oct 30 21:45:18.394301 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.394160 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerStarted","Data":"3e5c72ea10c7db14437516c2ab842f57dca944dc6eb5cd32b8a6972bbdf50d25"} Oct 30 21:45:18.625695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.625639 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podStartSLOduration=1.625621336 podStartE2EDuration="1.625621336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:18.621233167 +0000 UTC m=+2032.603933585" watchObservedRunningTime="2026-10-30 21:45:18.625621336 +0000 UTC m=+2032.608321752" Oct 30 21:45:18.674110 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.674045 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:18.883948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.883924 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:18.914244 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.914217 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\" (UniqueName: \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\") pod \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " Oct 30 21:45:18.914417 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.914402 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\") pod \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\" (UID: \"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1\") " Oct 30 21:45:18.914743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.914716 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca" pod "f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" (UID: "f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:18.915007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.914992 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.916295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:18.916270 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5" pod "f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" (UID: "f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1"). InnerVolumeSpecName "kube-api-access-f72h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.046337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.035186 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f72h5\" (UniqueName: \"kubernetes.io/projected/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1-kube-api-access-f72h5\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.391927 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.391687 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:19.398471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.398436 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:19.409000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.408962 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" event={"ID":"f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1","Type":"ContainerDied","Data":"bfd0eab7ef911b72a433656e9f11530d61dc3033122a1211d31f72f6d73ccade"} Oct 30 21:45:19.409165 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.409034 4921 scope.go:122] "RemoveContainer" containerID="177b3188dcfa6ff294d80a9cf0a51e987ad634cce7e3b9184f00b2e35e396f22" Oct 30 21:45:19.409241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.409166 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6" Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.412811 4921 generic.go:356] "Generic (PLEG): container finished" podID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerID="d99e5bbcbf2370ec833252f754b7c82d7e547bf5bd478e5cfb04e49782fbe936" exitCode=60 Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.412886 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerDied","Data":"d99e5bbcbf2370ec833252f754b7c82d7e547bf5bd478e5cfb04e49782fbe936"} Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.413410 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.414245 4921 scope.go:122] "RemoveContainer" containerID="d99e5bbcbf2370ec833252f754b7c82d7e547bf5bd478e5cfb04e49782fbe936" Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:19.414609 4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx_odh-ai-gateway-infra(e7f1ade2-11af-4121-822b-9067b8740ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" Oct 30 21:45:19.417575 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.414690 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:45:19.445867 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.445800 4921 scope.go:122] "RemoveContainer" containerID="4ec0392caab16e3ee737abb1f89433c9bcf3d7fdc260308665aba6c6085d79c5" Oct 30 21:45:19.516628 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.516527 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:19.628820 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.628787 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:19.634716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:19.634688 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-be9451-29889945-ccsj6"] Oct 30 21:45:20.178768 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.178732 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" path="/var/lib/kubelet/pods/f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1/volumes" Oct 30 21:45:20.588861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.588836 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:20.682343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.680728 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\") pod \"e7f1ade2-11af-4121-822b-9067b8740ce4\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " Oct 30 21:45:20.682343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.680880 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\" (UniqueName: \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\") pod \"e7f1ade2-11af-4121-822b-9067b8740ce4\" (UID: \"e7f1ade2-11af-4121-822b-9067b8740ce4\") " Oct 30 21:45:20.682343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.681067 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca" pod "e7f1ade2-11af-4121-822b-9067b8740ce4" (UID: "e7f1ade2-11af-4121-822b-9067b8740ce4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.682343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.681407 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1ade2-11af-4121-822b-9067b8740ce4-maas-api-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.696689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.687816 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x" pod "e7f1ade2-11af-4121-822b-9067b8740ce4" (UID: "e7f1ade2-11af-4121-822b-9067b8740ce4"). InnerVolumeSpecName "kube-api-access-k9l5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.782505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:20.782429 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9l5x\" (UniqueName: \"kubernetes.io/projected/e7f1ade2-11af-4121-822b-9067b8740ce4-kube-api-access-k9l5x\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.434973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:21.434947 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" Oct 30 21:45:21.435165 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:21.435066 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx" event={"ID":"e7f1ade2-11af-4121-822b-9067b8740ce4","Type":"ContainerDied","Data":"19c9ecc8139794ffe557139cea2e7eff6c25c03973cef248214c9d9bc202c943"} Oct 30 21:45:21.435165 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:21.435118 4921 scope.go:122] "RemoveContainer" containerID="d99e5bbcbf2370ec833252f754b7c82d7e547bf5bd478e5cfb04e49782fbe936" Oct 30 21:45:21.599062 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:21.599027 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:21.650131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:21.650099 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5f31bd-29889945-snxtx"] Oct 30 21:45:22.168709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:22.168680 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" path="/var/lib/kubelet/pods/e7f1ade2-11af-4121-822b-9067b8740ce4/volumes" Oct 30 21:45:30.083714 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.083678 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:45:30.084143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.083905 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" podUID="11ca0026-ae00-428e-82ce-b92bfaa8db95" containerName="maas-api" containerID="cri-o://a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5" gracePeriod=30 Oct 30 21:45:30.396563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.396539 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:45:30.451085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451056 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\") pod \"11ca0026-ae00-428e-82ce-b92bfaa8db95\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " Oct 30 21:45:30.451226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451197 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\" (UniqueName: \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\") pod \"11ca0026-ae00-428e-82ce-b92bfaa8db95\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " Oct 30 21:45:30.451305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451292 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\") pod \"11ca0026-ae00-428e-82ce-b92bfaa8db95\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " Oct 30 21:45:30.451380 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451341 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") pod \"11ca0026-ae00-428e-82ce-b92bfaa8db95\" (UID: \"11ca0026-ae00-428e-82ce-b92bfaa8db95\") " Oct 30 21:45:30.451444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451400 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca" pod "11ca0026-ae00-428e-82ce-b92bfaa8db95" (UID: "11ca0026-ae00-428e-82ce-b92bfaa8db95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:30.451665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.451645 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11ca0026-ae00-428e-82ce-b92bfaa8db95-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:30.453206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.453185 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp" pod "11ca0026-ae00-428e-82ce-b92bfaa8db95" (UID: "11ca0026-ae00-428e-82ce-b92bfaa8db95"). InnerVolumeSpecName "kube-api-access-gr5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:30.453349 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.453306 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs" pod "11ca0026-ae00-428e-82ce-b92bfaa8db95" (UID: "11ca0026-ae00-428e-82ce-b92bfaa8db95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:30.453509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.453493 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls" pod "11ca0026-ae00-428e-82ce-b92bfaa8db95" (UID: "11ca0026-ae00-428e-82ce-b92bfaa8db95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:30.511451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.511411 4921 generic.go:356] "Generic (PLEG): container finished" podID="11ca0026-ae00-428e-82ce-b92bfaa8db95" containerID="a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5" exitCode=0 Oct 30 21:45:30.511696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.511458 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" Oct 30 21:45:30.511696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.511460 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" event={"ID":"11ca0026-ae00-428e-82ce-b92bfaa8db95","Type":"ContainerDied","Data":"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5"} Oct 30 21:45:30.511696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.511599 4921 scope.go:122] "RemoveContainer" containerID="a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5" Oct 30 21:45:30.511696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.511596 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr" event={"ID":"11ca0026-ae00-428e-82ce-b92bfaa8db95","Type":"ContainerDied","Data":"8fefd691be071516bd8265dd53dd0b7faf318cf0473ab9364243c7258df3f639"} Oct 30 21:45:30.543798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.543718 4921 scope.go:122] "RemoveContainer" containerID="a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5" Oct 30 21:45:30.544187 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:30.544163 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5\": container with ID starting with a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5 not found: ID does not exist" containerID="a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5" Oct 30 21:45:30.544288 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.544194 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5"} err="failed to get container status \"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5\": rpc error: code = NotFound desc = could not find container \"a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5\": container with ID starting with a3e8c8f43f0e0878a2e9394b642026e76119028a626e4fde11e7d794826f38d5 not found: ID does not exist" Oct 30 21:45:30.553492 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.553462 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:30.553600 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.553495 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11ca0026-ae00-428e-82ce-b92bfaa8db95-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:30.553600 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.553511 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5sp\" (UniqueName: \"kubernetes.io/projected/11ca0026-ae00-428e-82ce-b92bfaa8db95-kube-api-access-gr5sp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:30.687069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.686983 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:45:30.691108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:30.691071 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7cf226-594f9bbd8f-t7jrr"] Oct 30 21:45:32.177980 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:32.177941 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ca0026-ae00-428e-82ce-b92bfaa8db95" path="/var/lib/kubelet/pods/11ca0026-ae00-428e-82ce-b92bfaa8db95/volumes" Oct 30 21:45:37.540590 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:45:37.540547 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:45:42.747709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.747677 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:45:42.748282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748270 4921 cpu_manager.go:414] "containerMap: removing container" podUID="11ca0026-ae00-428e-82ce-b92bfaa8db95" containerName="maas-api" Oct 30 21:45:42.748351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748283 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ca0026-ae00-428e-82ce-b92bfaa8db95" containerName="maas-api" Oct 30 21:45:42.748351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748305 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748310 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748340 4921 cpu_manager.go:414] "containerMap: removing container" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748346 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748356 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748360 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748368 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748373 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748385 4921 cpu_manager.go:414] "containerMap: removing container" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748389 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748473 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748482 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748487 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748494 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748500 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748508 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ca0026-ae00-428e-82ce-b92bfaa8db95" containerName="maas-api" Oct 30 21:45:42.748509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.748514 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9342e918-b556-4b4b-83b1-b96a704566a2" containerName="cleanup" Oct 30 21:45:42.753459 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.753440 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.869175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.860945 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:45:42.884373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.882289 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.884755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.884727 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885044 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885249 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885511 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885491 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885672 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885834 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885806 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.885967 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.885956 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.886078 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.886068 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.888257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.888231 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9085f4-openshift-default-dockercfg-kx2k4\"" Oct 30 21:45:42.987971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.987931 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988004 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988061 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988084 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988114 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988167 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988200 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.988287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988248 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.991154 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988362 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.991154 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.988845 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.991154 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.989921 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.993892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.991970 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.993892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.992212 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.993892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.992482 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:42.996548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:42.996507 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:43.010381 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.009182 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:43.047003 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.046969 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:43.047230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.047138 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\") pod \"e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:43.067943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.067920 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:43.510906 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:45:43.510878 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858ec943_5d7f_429f_8de2_ef330e71578a.slice/crio-c4a9b7e0479b16f9e4a068c5df55de815ff814af4972398068641f7cedb392b1 WatchSource:0}: Error finding container c4a9b7e0479b16f9e4a068c5df55de815ff814af4972398068641f7cedb392b1: Status 404 returned error can't find the container with id c4a9b7e0479b16f9e4a068c5df55de815ff814af4972398068641f7cedb392b1 Oct 30 21:45:43.511201 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.511175 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:45:43.514373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.514297 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:43.514482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.514420 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:43.514482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.514465 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:45:43.625270 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.625237 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerStarted","Data":"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e"} Oct 30 21:45:43.625469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.625298 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerStarted","Data":"c4a9b7e0479b16f9e4a068c5df55de815ff814af4972398068641f7cedb392b1"} Oct 30 21:45:43.748814 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:43.748752 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podStartSLOduration=1.748731732 podStartE2EDuration="1.748731732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:43.741593314 +0000 UTC m=+2057.724293732" watchObservedRunningTime="2026-10-30 21:45:43.748731732 +0000 UTC m=+2057.731432152" Oct 30 21:45:44.068660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:44.068633 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:44.073773 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:44.073748 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:44.633524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:44.633490 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:44.634358 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:44.634307 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:45:59.409661 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:59.409626 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:45:59.410248 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:45:59.409853 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" containerID="cri-o://d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef" gracePeriod=30 Oct 30 21:46:00.923107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:00.923074 4921 patch_prober.go:29] interesting pod/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:00.923529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:00.923144 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:03.887524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:03.887484 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:46:03.888028 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:03.887685 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerName="discovery" containerID="cri-o://5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6" gracePeriod=30 Oct 30 21:46:04.163685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.163663 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:46:04.313596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.313566 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.313780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.313603 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.313780 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.313738 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.313904 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.313841 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.314001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.313972 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.314202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.314092 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.314202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.314088 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.314202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.314181 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\") pod \"12540083-1aa3-49d1-9c19-41ff86e481a8\" (UID: \"12540083-1aa3-49d1-9c19-41ff86e481a8\") " Oct 30 21:46:04.314818 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.314784 4921 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-ca-configmap\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.316197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316159 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "kube-api-access-xnh4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:04.316365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316282 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:04.316445 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316422 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:04.316494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316442 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:04.316711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316692 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.316789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.316704 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig" pod "12540083-1aa3-49d1-9c19-41ff86e481a8" (UID: "12540083-1aa3-49d1-9c19-41ff86e481a8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415695 4921 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-cacerts\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415723 4921 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12540083-1aa3-49d1-9c19-41ff86e481a8-local-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415734 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415742 4921 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-kubeconfig\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415752 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh4w\" (UniqueName: \"kubernetes.io/projected/12540083-1aa3-49d1-9c19-41ff86e481a8-kube-api-access-xnh4w\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.415764 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.415761 4921 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12540083-1aa3-49d1-9c19-41ff86e481a8-istio-csr-dns-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.662586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.662561 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:46:04.819130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819100 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819138 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819188 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819276 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819337 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819297 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819369 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819401 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819448 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.819566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.819473 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\") pod \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\" (UID: \"b6bf56a4-259a-4327-9859-cf3e8bb85a26\") " Oct 30 21:46:04.821386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.820511 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.821386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.820820 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:04.821386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.821069 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.821950 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.821845 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.822310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.822291 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.822485 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.822420 4921 generic.go:356] "Generic (PLEG): container finished" podID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerID="5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6" exitCode=0 Oct 30 21:46:04.822608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.822544 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" Oct 30 21:46:04.824723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.824702 4921 generic.go:356] "Generic (PLEG): container finished" podID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerID="d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef" exitCode=0 Oct 30 21:46:04.824930 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.824916 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" Oct 30 21:46:04.825364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.822446 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" event={"ID":"12540083-1aa3-49d1-9c19-41ff86e481a8","Type":"ContainerDied","Data":"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6"} Oct 30 21:46:04.825554 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.825538 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf" event={"ID":"12540083-1aa3-49d1-9c19-41ff86e481a8","Type":"ContainerDied","Data":"88f8c16fc6c5046683619f264413c6770174685f94c02d99d506620d87f7abd0"} Oct 30 21:46:04.825674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.825659 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" event={"ID":"b6bf56a4-259a-4327-9859-cf3e8bb85a26","Type":"ContainerDied","Data":"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef"} Oct 30 21:46:04.825791 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.825778 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9" event={"ID":"b6bf56a4-259a-4327-9859-cf3e8bb85a26","Type":"ContainerDied","Data":"9cd2f6f6615cda7dffb556d48fbc76909759fc38d6b935bb74759acd7b2e6261"} Oct 30 21:46:04.825888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.825789 4921 scope.go:122] "RemoveContainer" containerID="5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6" Oct 30 21:46:04.832810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.832760 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:04.832810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.832776 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:04.832987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.832963 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:04.835540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.835514 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw" pod "b6bf56a4-259a-4327-9859-cf3e8bb85a26" (UID: "b6bf56a4-259a-4327-9859-cf3e8bb85a26"). InnerVolumeSpecName "kube-api-access-8qfhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:04.841325 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.841296 4921 scope.go:122] "RemoveContainer" containerID="5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6" Oct 30 21:46:04.841729 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:04.841703 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6\": container with ID starting with 5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6 not found: ID does not exist" containerID="5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6" Oct 30 21:46:04.841812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.841740 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6"} err="failed to get container status \"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6\": rpc error: code = NotFound desc = could not find container \"5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6\": container with ID starting with 5000f21b2597442e7cbe7c82eeb07c2dd13669216ee8867f7f5f9308f6d095a6 not found: ID does not exist" Oct 30 21:46:04.841812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.841763 4921 scope.go:122] "RemoveContainer" containerID="d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef" Oct 30 21:46:04.854464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.854438 4921 scope.go:122] "RemoveContainer" containerID="d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef" Oct 30 21:46:04.854760 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:04.854734 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef\": container with ID starting with d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef not found: ID does not exist" containerID="d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef" Oct 30 21:46:04.854861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.854769 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef"} err="failed to get container status \"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef\": rpc error: code = NotFound desc = could not find container \"d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef\": container with ID starting with d2016db59308537dd1074cb06082e6ac621da5486f241aea8f1c6a324fdb86ef not found: ID does not exist" Oct 30 21:46:04.920902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920878 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.920902 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920902 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920911 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920919 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920927 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920936 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920945 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bf56a4-259a-4327-9859-cf3e8bb85a26-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920952 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bf56a4-259a-4327-9859-cf3e8bb85a26-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.920961 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/b6bf56a4-259a-4327-9859-cf3e8bb85a26-kube-api-access-8qfhw\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:04.995411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:04.995378 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:46:05.006007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:05.005975 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x6fgf"] Oct 30 21:46:05.251013 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:05.250972 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:46:05.254581 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:05.254556 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-q6jc9"] Oct 30 21:46:06.168759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:06.168728 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" path="/var/lib/kubelet/pods/12540083-1aa3-49d1-9c19-41ff86e481a8/volumes" Oct 30 21:46:06.169235 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:06.169221 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" path="/var/lib/kubelet/pods/b6bf56a4-259a-4327-9859-cf3e8bb85a26/volumes" Oct 30 21:46:07.542007 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:07.541968 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:46:09.432084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:09.432051 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:46:09.432494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:09.432287 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" containerID="cri-o://4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa" gracePeriod=30 Oct 30 21:46:14.129971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.129936 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:14.130368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.129995 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:14.699483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.699463 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:46:14.747666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747644 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.747781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747681 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.747781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747770 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.747872 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747834 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.747925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747877 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.747925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747910 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.748031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.747941 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.748094 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748006 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.748094 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748017 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.748196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748119 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\") pod \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\" (UID: \"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2\") " Oct 30 21:46:14.748196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748097 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.748196 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748184 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.748389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748237 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.748562 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748538 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:14.748678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748637 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.748678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748668 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.748803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748682 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.748803 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.748696 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.750074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.750044 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.750074 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.750068 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:14.750191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.750082 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:14.750563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.750542 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q" pod "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" (UID: "6973bd1c-bf6b-4653-8ee4-1eebe8234bf2"). InnerVolumeSpecName "kube-api-access-6z42q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:14.849961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.849937 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z42q\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-kube-api-access-6z42q\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.849961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.849961 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.850091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.849970 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.850091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.849979 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.850091 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.849987 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.908105 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.908078 4921 generic.go:356] "Generic (PLEG): container finished" podID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerID="4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa" exitCode=0 Oct 30 21:46:14.908241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.908111 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerDied","Data":"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa"} Oct 30 21:46:14.908241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.908131 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" Oct 30 21:46:14.908241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.908151 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254" event={"ID":"6973bd1c-bf6b-4653-8ee4-1eebe8234bf2","Type":"ContainerDied","Data":"8f1e378b8bc19bf0cc038b07b8f2387c3822f2fd5c2b5cd114ccc7607413124e"} Oct 30 21:46:14.908241 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.908172 4921 scope.go:122] "RemoveContainer" containerID="4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa" Oct 30 21:46:14.919878 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.919857 4921 scope.go:122] "RemoveContainer" containerID="9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e" Oct 30 21:46:14.929511 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.929494 4921 scope.go:122] "RemoveContainer" containerID="4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa" Oct 30 21:46:14.929755 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:14.929736 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa\": container with ID starting with 4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa not found: ID does not exist" containerID="4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa" Oct 30 21:46:14.929819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.929764 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa"} err="failed to get container status \"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa\": rpc error: code = NotFound desc = could not find container \"4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa\": container with ID starting with 4a8f3c64fe064bb45c7919786e4bf5569118aaeab35a3e1a3d89ec95a3e16dfa not found: ID does not exist" Oct 30 21:46:14.929819 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.929782 4921 scope.go:122] "RemoveContainer" containerID="9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e" Oct 30 21:46:14.930028 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:14.930009 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e\": container with ID starting with 9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e not found: ID does not exist" containerID="9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e" Oct 30 21:46:14.930072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:14.930033 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e"} err="failed to get container status \"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e\": rpc error: code = NotFound desc = could not find container \"9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e\": container with ID starting with 9e87b9d766f89a0dfe02c0d26b329faa3fb1d576c09774618ee544c8ad01bc7e not found: ID does not exist" Oct 30 21:46:15.071082 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.071025 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:46:15.075131 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.075113 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:46:15.079563 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.079531 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e0f967-openshift-default-67f666cdf-mn254"] Oct 30 21:46:15.120425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.120403 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:46:15.682675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.682638 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:46:15.683173 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683159 4921 cpu_manager.go:414] "containerMap: removing container" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:46:15.683229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683173 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9313f73-9a4e-4b98-a376-4fa2bb9ae7d1" containerName="cleanup" Oct 30 21:46:15.683229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683202 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" Oct 30 21:46:15.683229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683210 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" Oct 30 21:46:15.683229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683226 4921 cpu_manager.go:414] "containerMap: removing container" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerName="discovery" Oct 30 21:46:15.683229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683231 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerName="discovery" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683241 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683246 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683262 4921 cpu_manager.go:414] "containerMap: removing container" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683267 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683275 4921 cpu_manager.go:414] "containerMap: removing container" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683279 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683377 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683387 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683392 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f1ade2-11af-4121-822b-9067b8740ce4" containerName="cleanup" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683401 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf56a4-259a-4327-9859-cf3e8bb85a26" containerName="istio-proxy" Oct 30 21:46:15.683489 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.683409 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="12540083-1aa3-49d1-9c19-41ff86e481a8" containerName="discovery" Oct 30 21:46:15.688351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.688304 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.744270 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.744244 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:46:15.758811 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.758781 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr28\" (UniqueName: \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.758945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.758836 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.758945 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.758887 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.759111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.759094 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.781865 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.781840 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3ab88248-serving-cert\"" Oct 30 21:46:15.860225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.860196 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.860384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.860241 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr28\" (UniqueName: \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.860384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.860286 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.860384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.860342 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.860552 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:15.860491 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-serving-cert: secret "maas-api-e2e-mt-3ab88248-serving-cert" not found Oct 30 21:46:15.860613 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:15.860568 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls podName:08d45e4b-7dcc-4312-9df0-f3f7b4ae6149 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:16.360548406 +0000 UTC m=+2090.343248819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls") pod "maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" (UID: "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149") : secret "maas-api-e2e-mt-3ab88248-serving-cert" not found Oct 30 21:46:15.861073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.861047 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.862908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.862888 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:15.917252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:15.917224 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr28\" (UniqueName: \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:16.170192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:16.170156 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6973bd1c-bf6b-4653-8ee4-1eebe8234bf2" path="/var/lib/kubelet/pods/6973bd1c-bf6b-4653-8ee4-1eebe8234bf2/volumes" Oct 30 21:46:16.365810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:16.365777 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:16.368161 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:16.368131 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:16.602217 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:16.602171 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:16.982102 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:16.982071 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:46:16.982514 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:46:16.982469 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d45e4b_7dcc_4312_9df0_f3f7b4ae6149.slice/crio-6ac86252ca438f1a1ef572efa09076ee1d8a3db901a99dd4510d9319f208f95d WatchSource:0}: Error finding container 6ac86252ca438f1a1ef572efa09076ee1d8a3db901a99dd4510d9319f208f95d: Status 404 returned error can't find the container with id 6ac86252ca438f1a1ef572efa09076ee1d8a3db901a99dd4510d9319f208f95d Oct 30 21:46:17.930850 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:17.930821 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" event={"ID":"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149","Type":"ContainerStarted","Data":"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3"} Oct 30 21:46:17.930975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:17.930895 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" event={"ID":"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149","Type":"ContainerStarted","Data":"6ac86252ca438f1a1ef572efa09076ee1d8a3db901a99dd4510d9319f208f95d"} Oct 30 21:46:17.930975 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:17.930930 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:18.058530 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:18.058487 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" podStartSLOduration=2.220578932 podStartE2EDuration="3.058474538s" totalImagesPullingTime="837.895606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.055377399 +0000 UTC m=+2092.038077818" watchObservedRunningTime="2026-10-30 21:46:18.058474538 +0000 UTC m=+2092.041174955" Oct 30 21:46:22.524925 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:22.524886 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:46:23.945441 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:23.945413 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:46:26.759048 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:26.759016 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:46:26.771146 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:26.771126 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:46:26.813215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:26.813192 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:46:26.825794 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:26.825770 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:46:27.526528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:27.526476 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" containerID="cri-o://e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479" gracePeriod=25 Oct 30 21:46:30.094855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.094814 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" containerID="cri-o://c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c" gracePeriod=45 Oct 30 21:46:30.144784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.144748 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" containerID="cri-o://ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f" gracePeriod=45 Oct 30 21:46:30.490225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.490202 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:46:30.493571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.493552 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:46:30.570125 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570089 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.570330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570147 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.570330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570193 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\" (UniqueName: \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570327 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570464 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570394 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570461 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570505 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.570568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570535 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570782 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570620 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\" (UniqueName: \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\") pod \"d572a994-8a3a-425f-ac9a-51bb2a5016db\" (UID: \"d572a994-8a3a-425f-ac9a-51bb2a5016db\") " Oct 30 21:46:30.570782 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570674 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.570782 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570705 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.570938 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.570791 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\" (UniqueName: \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\") pod \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\" (UID: \"fd8d14b8-f2d6-4076-907f-dd31236d9fc5\") " Oct 30 21:46:30.571262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.571237 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.571449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.571428 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.572008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.571853 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.572008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.571924 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.572008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.571978 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.572491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.572456 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.572964 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.572993 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.573006 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.573021 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.573034 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.573066 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.573046 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.574643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.574585 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.574798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.574762 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:30.575792 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.575769 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "kube-api-access-lm8s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:30.575876 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.575812 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "kube-api-access-4t5tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:30.576786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.576756 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs" pod "fd8d14b8-f2d6-4076-907f-dd31236d9fc5" (UID: "fd8d14b8-f2d6-4076-907f-dd31236d9fc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:30.576950 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.576931 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm" pod "d572a994-8a3a-425f-ac9a-51bb2a5016db" (UID: "d572a994-8a3a-425f-ac9a-51bb2a5016db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674637 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d572a994-8a3a-425f-ac9a-51bb2a5016db-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674667 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674680 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5tb\" (UniqueName: \"kubernetes.io/projected/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-kube-api-access-4t5tb\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674694 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8d14b8-f2d6-4076-907f-dd31236d9fc5-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674706 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8s6\" (UniqueName: \"kubernetes.io/projected/d572a994-8a3a-425f-ac9a-51bb2a5016db-kube-api-access-lm8s6\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:30.674716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:30.674717 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d572a994-8a3a-425f-ac9a-51bb2a5016db-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:31.052561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.052522 4921 generic.go:356] "Generic (PLEG): container finished" podID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerID="c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c" exitCode=0 Oct 30 21:46:31.052801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.052587 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" Oct 30 21:46:31.052801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.052606 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" event={"ID":"fd8d14b8-f2d6-4076-907f-dd31236d9fc5","Type":"ContainerDied","Data":"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c"} Oct 30 21:46:31.052801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.052665 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7" event={"ID":"fd8d14b8-f2d6-4076-907f-dd31236d9fc5","Type":"ContainerDied","Data":"98b7f000fdc750ea167972f39785e071eb9a6978d4940bbb2eced6e825a65d26"} Oct 30 21:46:31.052801 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.052682 4921 scope.go:122] "RemoveContainer" containerID="c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c" Oct 30 21:46:31.054348 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.054306 4921 generic.go:356] "Generic (PLEG): container finished" podID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerID="ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f" exitCode=0 Oct 30 21:46:31.054495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.054360 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" event={"ID":"d572a994-8a3a-425f-ac9a-51bb2a5016db","Type":"ContainerDied","Data":"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f"} Oct 30 21:46:31.054495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.054398 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" event={"ID":"d572a994-8a3a-425f-ac9a-51bb2a5016db","Type":"ContainerDied","Data":"8654035f11d053b0da052f5d62759fe23cf1f04e5942dd816cf3f568eb59ac0d"} Oct 30 21:46:31.054495 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.054373 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" Oct 30 21:46:31.063842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.063826 4921 scope.go:122] "RemoveContainer" containerID="c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c" Oct 30 21:46:31.064180 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:31.064082 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c\": container with ID starting with c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c not found: ID does not exist" containerID="c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c" Oct 30 21:46:31.064180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.064106 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c"} err="failed to get container status \"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c\": rpc error: code = NotFound desc = could not find container \"c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c\": container with ID starting with c36491d8e62741f5e8bcc1f528a07ab80c2ffdc99ffcb906b9564ae95b07ff4c not found: ID does not exist" Oct 30 21:46:31.064180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.064122 4921 scope.go:122] "RemoveContainer" containerID="ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f" Oct 30 21:46:31.072559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.072542 4921 scope.go:122] "RemoveContainer" containerID="ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f" Oct 30 21:46:31.072822 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:31.072804 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f\": container with ID starting with ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f not found: ID does not exist" containerID="ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f" Oct 30 21:46:31.072886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.072831 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f"} err="failed to get container status \"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f\": rpc error: code = NotFound desc = could not find container \"ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f\": container with ID starting with ce69410530d2ad8e4b576b6bb428f1083d443ec712c03e6f9570da14d372e94f not found: ID does not exist" Oct 30 21:46:31.233684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.233655 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:46:31.237424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.237387 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv"] Oct 30 21:46:31.386225 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.386040 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:46:31.389425 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.389399 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f52c3/distinct-2-w3-9f52c3-kserve-86464777bc-kdns7"] Oct 30 21:46:31.487268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:31.487236 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f52c3/premium-w3-9f52c3-kserve-68dd9888f-cppbv" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:46:32.168783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:32.168755 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" path="/var/lib/kubelet/pods/d572a994-8a3a-425f-ac9a-51bb2a5016db/volumes" Oct 30 21:46:32.169182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:32.169168 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" path="/var/lib/kubelet/pods/fd8d14b8-f2d6-4076-907f-dd31236d9fc5/volumes" Oct 30 21:46:34.094550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.094509 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:46:34.379088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.378101 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:46:34.379088 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.378455 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" podUID="c56e56e0-e586-4f59-afc2-7c10885a6535" containerName="maas-api" containerID="cri-o://a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d" gracePeriod=30 Oct 30 21:46:34.681654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.681623 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:46:34.826741 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.826709 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\" (UniqueName: \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\") pod \"c56e56e0-e586-4f59-afc2-7c10885a6535\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " Oct 30 21:46:34.826923 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.826804 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") pod \"c56e56e0-e586-4f59-afc2-7c10885a6535\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " Oct 30 21:46:34.826923 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.826891 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\") pod \"c56e56e0-e586-4f59-afc2-7c10885a6535\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " Oct 30 21:46:34.827041 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.826943 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\") pod \"c56e56e0-e586-4f59-afc2-7c10885a6535\" (UID: \"c56e56e0-e586-4f59-afc2-7c10885a6535\") " Oct 30 21:46:34.827736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.827709 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca" pod "c56e56e0-e586-4f59-afc2-7c10885a6535" (UID: "c56e56e0-e586-4f59-afc2-7c10885a6535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:34.829487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.829442 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls" pod "c56e56e0-e586-4f59-afc2-7c10885a6535" (UID: "c56e56e0-e586-4f59-afc2-7c10885a6535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:34.830724 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.830702 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs" pod "c56e56e0-e586-4f59-afc2-7c10885a6535" (UID: "c56e56e0-e586-4f59-afc2-7c10885a6535"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:34.832162 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.832137 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62" pod "c56e56e0-e586-4f59-afc2-7c10885a6535" (UID: "c56e56e0-e586-4f59-afc2-7c10885a6535"). InnerVolumeSpecName "kube-api-access-4pc62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:34.931580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.928482 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pc62\" (UniqueName: \"kubernetes.io/projected/c56e56e0-e586-4f59-afc2-7c10885a6535-kube-api-access-4pc62\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:34.931580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.928528 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:34.931580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.928545 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56e56e0-e586-4f59-afc2-7c10885a6535-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:34.931580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:34.928559 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56e56e0-e586-4f59-afc2-7c10885a6535-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:35.090827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.090793 4921 generic.go:356] "Generic (PLEG): container finished" podID="c56e56e0-e586-4f59-afc2-7c10885a6535" containerID="a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d" exitCode=0 Oct 30 21:46:35.090989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.090854 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" Oct 30 21:46:35.090989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.090875 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" event={"ID":"c56e56e0-e586-4f59-afc2-7c10885a6535","Type":"ContainerDied","Data":"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d"} Oct 30 21:46:35.090989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.090933 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd" event={"ID":"c56e56e0-e586-4f59-afc2-7c10885a6535","Type":"ContainerDied","Data":"7de87a3371640286c3745e9d92c684fe26cae2379dc9f3348a352bbab036a641"} Oct 30 21:46:35.090989 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.090961 4921 scope.go:122] "RemoveContainer" containerID="a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d" Oct 30 21:46:35.104660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.104638 4921 scope.go:122] "RemoveContainer" containerID="a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d" Oct 30 21:46:35.105003 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:35.104979 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d\": container with ID starting with a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d not found: ID does not exist" containerID="a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d" Oct 30 21:46:35.105055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.105014 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d"} err="failed to get container status \"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d\": rpc error: code = NotFound desc = could not find container \"a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d\": container with ID starting with a22c494be9d41c0d1aec7421f1b6f69b1d77badbc79c8f2aaa60e59a25c3309d not found: ID does not exist" Oct 30 21:46:35.256242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.256210 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:46:35.261658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.261633 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-815dab-6999db7887-wzbtd"] Oct 30 21:46:35.454475 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.454431 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:46:35.454662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:35.454481 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:46:36.170008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:36.169977 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56e56e0-e586-4f59-afc2-7c10885a6535" path="/var/lib/kubelet/pods/c56e56e0-e586-4f59-afc2-7c10885a6535/volumes" Oct 30 21:46:37.108548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.108521 4921 generic.go:356] "Generic (PLEG): container finished" podID="8b07a77a-abde-4656-a9cb-539187d89e50" containerID="f411ff91e86e6845834c50379174f3b64b7504fc2511145d7dfa7913af0354da" exitCode=0 Oct 30 21:46:37.108786 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.108598 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerDied","Data":"f411ff91e86e6845834c50379174f3b64b7504fc2511145d7dfa7913af0354da"} Oct 30 21:46:37.109010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.108994 4921 scope.go:122] "RemoveContainer" containerID="f411ff91e86e6845834c50379174f3b64b7504fc2511145d7dfa7913af0354da" Oct 30 21:46:37.110427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.110396 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:37.110494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.110468 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:37.110526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:37.110503 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:46:37.543559 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:37.543514 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:46:38.131135 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:38.131103 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerStarted","Data":"d772a6e33ca5df4c979181f37a676e45a3c879b06063f1abfefbbf2e195a6e47"} Oct 30 21:46:38.391506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:38.391425 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:46:38.391660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:38.391542 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:46:38.391660 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:38.391582 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:46:39.391180 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:39.391147 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:46:39.391601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:39.391195 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:46:40.391168 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.391135 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:46:40.391339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.391184 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:46:40.936366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.936308 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:46:40.937005 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.936986 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c56e56e0-e586-4f59-afc2-7c10885a6535" containerName="maas-api" Oct 30 21:46:40.937109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937006 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56e56e0-e586-4f59-afc2-7c10885a6535" containerName="maas-api" Oct 30 21:46:40.937109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937025 4921 cpu_manager.go:414] "containerMap: removing container" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" Oct 30 21:46:40.937109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937030 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" Oct 30 21:46:40.937109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937039 4921 cpu_manager.go:414] "containerMap: removing container" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" Oct 30 21:46:40.937109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937045 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" Oct 30 21:46:40.937397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937135 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56e56e0-e586-4f59-afc2-7c10885a6535" containerName="maas-api" Oct 30 21:46:40.937397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937147 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d572a994-8a3a-425f-ac9a-51bb2a5016db" containerName="main" Oct 30 21:46:40.937397 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.937155 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8d14b8-f2d6-4076-907f-dd31236d9fc5" containerName="main" Oct 30 21:46:40.940531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:40.940511 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.044136 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.044108 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9085f4-serving-cert\"" Oct 30 21:46:41.047779 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.047749 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:46:41.138578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.138542 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.138766 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.138683 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4sx\" (UniqueName: \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.138840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.138788 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.138892 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.138848 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.241335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.240073 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4sx\" (UniqueName: \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.241335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.240193 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.241335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.240254 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.241335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.240417 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.242069 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:41.242054 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-serving-cert: secret "maas-api-e2e-shared-b-w0-9085f4-serving-cert" not found Oct 30 21:46:41.242240 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:41.242230 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls podName:880b8389-ecbc-4dbc-99ae-9af514cda5ae nodeName:}" failed. No retries permitted until 2026-10-30 21:46:41.742215362 +0000 UTC m=+2115.724915759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls") pod "maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" (UID: "880b8389-ecbc-4dbc-99ae-9af514cda5ae") : secret "maas-api-e2e-shared-b-w0-9085f4-serving-cert" not found Oct 30 21:46:41.242394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.242370 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.244170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.244149 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.297037 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.297012 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4sx\" (UniqueName: \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.395939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.395894 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:46:41.396448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.396422 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:46:41.745652 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.745617 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.748188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.748156 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:41.854431 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:41.854396 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:42.261195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:42.261169 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:46:42.264392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:42.264373 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:46:42.391175 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:42.391145 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:46:42.391335 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:42.391204 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 21:46:43.166070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.166041 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" event={"ID":"880b8389-ecbc-4dbc-99ae-9af514cda5ae","Type":"ContainerStarted","Data":"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190"} Oct 30 21:46:43.166398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.166095 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" event={"ID":"880b8389-ecbc-4dbc-99ae-9af514cda5ae","Type":"ContainerStarted","Data":"091a2b6c98cb3029f995ca88ea6f550e2dde1e7669c1fa695f63c5ba954b51c8"} Oct 30 21:46:43.166398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.166128 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:43.289904 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.289863 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" podStartSLOduration=2.491439399 podStartE2EDuration="3.289850549s" totalImagesPullingTime="798.41115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:43.286175482 +0000 UTC m=+2117.268875911" watchObservedRunningTime="2026-10-30 21:46:43.289850549 +0000 UTC m=+2117.272550990" Oct 30 21:46:43.395493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.395428 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:46:43.395922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.395905 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:46:43.396626 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:43.396599 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:46:44.094804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:44.094770 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:46:47.830480 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.830454 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:46:47.920519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.920493 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\") pod \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " Oct 30 21:46:47.920676 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.920546 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\" (UniqueName: \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\") pod \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\" (UID: \"3fb7e337-dc8d-4827-a623-3e3eba0902fe\") " Oct 30 21:46:47.920882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.920859 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume" pod "3fb7e337-dc8d-4827-a623-3e3eba0902fe" (UID: "3fb7e337-dc8d-4827-a623-3e3eba0902fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:47.921064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.921042 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb7e337-dc8d-4827-a623-3e3eba0902fe-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:47.922548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:47.922531 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx" pod "3fb7e337-dc8d-4827-a623-3e3eba0902fe" (UID: "3fb7e337-dc8d-4827-a623-3e3eba0902fe"). InnerVolumeSpecName "kube-api-access-xgxvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:48.021919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.021890 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgxvx\" (UniqueName: \"kubernetes.io/projected/3fb7e337-dc8d-4827-a623-3e3eba0902fe-kube-api-access-xgxvx\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:48.200303 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.200270 4921 generic.go:356] "Generic (PLEG): container finished" podID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerID="e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479" exitCode=0 Oct 30 21:46:48.200470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.200354 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" Oct 30 21:46:48.200470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.200359 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" event={"ID":"3fb7e337-dc8d-4827-a623-3e3eba0902fe","Type":"ContainerDied","Data":"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479"} Oct 30 21:46:48.200470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.200417 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6" event={"ID":"3fb7e337-dc8d-4827-a623-3e3eba0902fe","Type":"ContainerDied","Data":"82cfa428ae2640c64b8bdedd88bef75ca672e3f4ab4af9957e892d218a3eb812"} Oct 30 21:46:48.200470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.200444 4921 scope.go:122] "RemoveContainer" containerID="e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479" Oct 30 21:46:48.210911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.210891 4921 scope.go:122] "RemoveContainer" containerID="e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479" Oct 30 21:46:48.211217 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:48.211199 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479\": container with ID starting with e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479 not found: ID does not exist" containerID="e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479" Oct 30 21:46:48.211309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.211223 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479"} err="failed to get container status \"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479\": rpc error: code = NotFound desc = could not find container \"e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479\": container with ID starting with e9f703beee8046afb38715b16be2b1a005dffa3209202e4b65782fa21ac65479 not found: ID does not exist" Oct 30 21:46:48.362550 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.362525 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:46:48.368395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:48.368372 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9f52c3-7f7767bdcc-t4kz6"] Oct 30 21:46:49.179132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:49.179102 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:46:50.169835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:50.169801 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" path="/var/lib/kubelet/pods/3fb7e337-dc8d-4827-a623-3e3eba0902fe/volumes" Oct 30 21:46:53.877682 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:53.877648 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:46:53.878089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:53.877850 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" containerID="cri-o://0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad" gracePeriod=30 Oct 30 21:46:55.621486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:55.621451 4921 patch_prober.go:29] interesting pod/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:55.622044 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:55.622014 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:59.169505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.169482 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:46:59.278438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.278404 4921 generic.go:356] "Generic (PLEG): container finished" podID="8d7bb186-b1d7-461f-a323-e3b945404994" containerID="0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad" exitCode=0 Oct 30 21:46:59.278585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.278452 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" Oct 30 21:46:59.278585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.278485 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerDied","Data":"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad"} Oct 30 21:46:59.278585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.278540 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28" event={"ID":"8d7bb186-b1d7-461f-a323-e3b945404994","Type":"ContainerDied","Data":"ed080986706fda104265c36388c4a9af455131f45a6abf24ea5d3fa27adda137"} Oct 30 21:46:59.278585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.278565 4921 scope.go:122] "RemoveContainer" containerID="0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad" Oct 30 21:46:59.288416 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.288397 4921 scope.go:122] "RemoveContainer" containerID="e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7" Oct 30 21:46:59.297845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.297828 4921 scope.go:122] "RemoveContainer" containerID="0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad" Oct 30 21:46:59.298085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298066 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298164 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:59.298093 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad\": container with ID starting with 0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad not found: ID does not exist" containerID="0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad" Oct 30 21:46:59.298164 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298131 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298164 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298129 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad"} err="failed to get container status \"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad\": rpc error: code = NotFound desc = could not find container \"0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad\": container with ID starting with 0a3a2fa4cb35d5c2b9d3a8cbe7a5384f624cd006f4cd44a93aed726f47c303ad not found: ID does not exist" Oct 30 21:46:59.298164 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298152 4921 scope.go:122] "RemoveContainer" containerID="e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7" Oct 30 21:46:59.298390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298168 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298227 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298287 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298335 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298379 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298437 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:59.298638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298443 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298573 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\") pod \"8d7bb186-b1d7-461f-a323-e3b945404994\" (UID: \"8d7bb186-b1d7-461f-a323-e3b945404994\") " Oct 30 21:46:59.298973 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:46:59.298899 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7\": container with ID starting with e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7 not found: ID does not exist" containerID="e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7" Oct 30 21:46:59.298973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.298936 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7"} err="failed to get container status \"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7\": rpc error: code = NotFound desc = could not find container \"e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7\": container with ID starting with e08a8b5b7139ea79a2e34d35498158b0cbc0b0bbd657590da7053d087b2806c7 not found: ID does not exist" Oct 30 21:46:59.299118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.299048 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:59.299118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.299064 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:59.299493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.299471 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.299568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.299498 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7bb186-b1d7-461f-a323-e3b945404994-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.299568 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.299513 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.301215 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.301188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "kube-api-access-bf8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:59.301506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.301487 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:59.302129 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.302102 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:59.302377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.302356 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:59.302451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.302400 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:59.303430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.303402 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data" pod "8d7bb186-b1d7-461f-a323-e3b945404994" (UID: "8d7bb186-b1d7-461f-a323-e3b945404994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400933 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400960 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400970 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400979 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8mz\" (UniqueName: \"kubernetes.io/projected/8d7bb186-b1d7-461f-a323-e3b945404994-kube-api-access-bf8mz\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400988 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7bb186-b1d7-461f-a323-e3b945404994-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.401007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.400996 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7bb186-b1d7-461f-a323-e3b945404994-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:59.705132 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.705097 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:46:59.752729 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:46:59.752698 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-815dab-openshift-default-9d5585597-cgs28"] Oct 30 21:47:00.169427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:00.169346 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" path="/var/lib/kubelet/pods/8d7bb186-b1d7-461f-a323-e3b945404994/volumes" Oct 30 21:47:00.676469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:00.676428 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:00.677002 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:00.676500 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:03.313566 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.313534 4921 generic.go:356] "Generic (PLEG): container finished" podID="858ec943-5d7f-429f-8de2-ef330e71578a" containerID="7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e" exitCode=0 Oct 30 21:47:03.313986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.313606 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerDied","Data":"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e"} Oct 30 21:47:03.313986 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.313967 4921 scope.go:122] "RemoveContainer" containerID="7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e" Oct 30 21:47:03.315345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.315299 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:03.315460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.315380 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:03.315460 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:03.315409 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:04.331187 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:04.331144 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerStarted","Data":"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be"} Oct 30 21:47:05.068229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:05.068192 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:47:05.068420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:05.068354 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:47:05.068420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:05.068398 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:47:06.069621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:06.069582 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:47:06.070018 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:06.069644 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:47:07.068582 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:07.068546 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:47:07.068762 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:07.068618 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:47:07.551051 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:47:07.551016 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:47:08.069060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:08.069022 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:47:08.069234 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:08.069074 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:47:09.068450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:09.068413 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:47:09.068846 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:09.068462 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:47:10.072194 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:10.072163 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:47:10.072614 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:10.072598 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:47:10.073207 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:10.073188 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:47:33.454755 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.454718 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:47:33.455259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455246 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:33.455300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455260 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:33.455300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455291 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" Oct 30 21:47:33.455300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455296 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" Oct 30 21:47:33.455433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455382 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:33.455433 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.455392 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb7e337-dc8d-4827-a623-3e3eba0902fe" containerName="payload-processing" Oct 30 21:47:33.460206 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.460188 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498457 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498426 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498557 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498616 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498601 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498630 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498690 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.498860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.498786 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6qf\" (UniqueName: \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.517261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.517235 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:47:33.561531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.561503 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3ab88248\"/\"default-dockercfg-2rkd4\"" Oct 30 21:47:33.562248 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.562227 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3ab88248\"/\"kube-root-ca.crt\"" Oct 30 21:47:33.562385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.562311 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3ab88248\"/\"openshift-service-ca.crt\"" Oct 30 21:47:33.562385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.562372 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3ab88248\"/\"e2e-lifecycle-model-3ab88248-kserve-self-signed-certs\"" Oct 30 21:47:33.602412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.602381 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.603105 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.603080 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.603809 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.603787 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.604330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.604294 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.604860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.604841 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.617407 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.617378 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6qf\" (UniqueName: \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.618145 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.616372 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.618338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.604744 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.618540 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.604233 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.622087 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.622031 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.633812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.633781 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.674001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.673972 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6qf\" (UniqueName: \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\") pod \"e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:33.772789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:33.772728 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:34.152971 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:34.152941 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:47:34.153980 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:47:34.153950 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55f436a8_12af_44e0_9d74_95575a49155d.slice/crio-3efb7b0a4e42d861f7172f4f30ddefabfab40670887cc8a698ad9686d2bd7394 WatchSource:0}: Error finding container 3efb7b0a4e42d861f7172f4f30ddefabfab40670887cc8a698ad9686d2bd7394: Status 404 returned error can't find the container with id 3efb7b0a4e42d861f7172f4f30ddefabfab40670887cc8a698ad9686d2bd7394 Oct 30 21:47:34.535500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:34.535448 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" event={"ID":"55f436a8-12af-44e0-9d74-95575a49155d","Type":"ContainerStarted","Data":"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d"} Oct 30 21:47:34.535961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:34.535507 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" event={"ID":"55f436a8-12af-44e0-9d74-95575a49155d","Type":"ContainerStarted","Data":"3efb7b0a4e42d861f7172f4f30ddefabfab40670887cc8a698ad9686d2bd7394"} Oct 30 21:47:34.650640 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:34.650545 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" podStartSLOduration=1.650528403 podStartE2EDuration="1.650528403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:34.650244031 +0000 UTC m=+2168.632944449" watchObservedRunningTime="2026-10-30 21:47:34.650528403 +0000 UTC m=+2168.633228822" Oct 30 21:47:34.783403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:34.783019 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:37.541951 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:47:37.541904 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:47:43.773011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:43.772976 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:43.785028 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:43.785006 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:44.615236 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:44.615208 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:47:48.422191 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.422159 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:47:48.422736 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.422363 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" containerID="cri-o://66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407" gracePeriod=30 Oct 30 21:47:48.910866 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.910830 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7193-676f26"] Oct 30 21:47:48.911432 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.911416 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:48.911493 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.911434 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:48.911580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.911570 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7bb186-b1d7-461f-a323-e3b945404994" containerName="istio-proxy" Oct 30 21:47:48.916141 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.916119 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-676f26" Oct 30 21:47:48.973274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:48.973243 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7193-676f26"] Oct 30 21:47:49.050120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.050088 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmshp\"" Oct 30 21:47:49.105404 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.105368 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qnw7\" (UniqueName: \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\") pod \"test-curl-7193-676f26\" (UID: \"276d2c75-39cf-489c-abfe-f9294dbc250d\") " pod="opendatahub/test-curl-7193-676f26" Oct 30 21:47:49.207552 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.207509 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qnw7\" (UniqueName: \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\") pod \"test-curl-7193-676f26\" (UID: \"276d2c75-39cf-489c-abfe-f9294dbc250d\") " pod="opendatahub/test-curl-7193-676f26" Oct 30 21:47:49.263451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.263418 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qnw7\" (UniqueName: \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\") pod \"test-curl-7193-676f26\" (UID: \"276d2c75-39cf-489c-abfe-f9294dbc250d\") " pod="opendatahub/test-curl-7193-676f26" Oct 30 21:47:49.535709 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.535637 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-676f26" Oct 30 21:47:49.911497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:49.911454 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7193-676f26"] Oct 30 21:47:49.913648 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:47:49.913603 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276d2c75_39cf_489c_abfe_f9294dbc250d.slice/crio-cb0bedaad40e96a55f5faea615d91bd35438c56c11b4add8e1b9a25b89c09469 WatchSource:0}: Error finding container cb0bedaad40e96a55f5faea615d91bd35438c56c11b4add8e1b9a25b89c09469: Status 404 returned error can't find the container with id cb0bedaad40e96a55f5faea615d91bd35438c56c11b4add8e1b9a25b89c09469 Oct 30 21:47:50.268539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.268512 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:47:50.268714 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.268687 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerName="maas-api" containerID="cri-o://35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd" gracePeriod=30 Oct 30 21:47:50.346928 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.346892 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.73:8443/health\": dial tcp 10.132.0.73:8443: connect: connection refused" Oct 30 21:47:50.542747 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.542672 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:47:50.692309 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.692272 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") pod \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " Oct 30 21:47:50.692478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.692429 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\") pod \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " Oct 30 21:47:50.692539 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.692511 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\" (UniqueName: \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\") pod \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " Oct 30 21:47:50.692591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.692548 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\") pod \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\" (UID: \"fc8e7af9-57c0-4000-8d8c-5ee30662aaef\") " Oct 30 21:47:50.692790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.692763 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca" pod "fc8e7af9-57c0-4000-8d8c-5ee30662aaef" (UID: "fc8e7af9-57c0-4000-8d8c-5ee30662aaef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:50.693053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.693035 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:50.695054 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.695028 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls" pod "fc8e7af9-57c0-4000-8d8c-5ee30662aaef" (UID: "fc8e7af9-57c0-4000-8d8c-5ee30662aaef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:50.695193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.695050 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9" pod "fc8e7af9-57c0-4000-8d8c-5ee30662aaef" (UID: "fc8e7af9-57c0-4000-8d8c-5ee30662aaef"). InnerVolumeSpecName "kube-api-access-kmbk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:50.695193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.695119 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs" pod "fc8e7af9-57c0-4000-8d8c-5ee30662aaef" (UID: "fc8e7af9-57c0-4000-8d8c-5ee30662aaef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:50.715033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.715000 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7193-676f26" event={"ID":"276d2c75-39cf-489c-abfe-f9294dbc250d","Type":"ContainerStarted","Data":"2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84"} Oct 30 21:47:50.715153 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.715052 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7193-676f26" event={"ID":"276d2c75-39cf-489c-abfe-f9294dbc250d","Type":"ContainerStarted","Data":"cb0bedaad40e96a55f5faea615d91bd35438c56c11b4add8e1b9a25b89c09469"} Oct 30 21:47:50.716325 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.716297 4921 generic.go:356] "Generic (PLEG): container finished" podID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerID="35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd" exitCode=0 Oct 30 21:47:50.716406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.716339 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" event={"ID":"fc8e7af9-57c0-4000-8d8c-5ee30662aaef","Type":"ContainerDied","Data":"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd"} Oct 30 21:47:50.716406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.716392 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" event={"ID":"fc8e7af9-57c0-4000-8d8c-5ee30662aaef","Type":"ContainerDied","Data":"f54d0714e8cb1071c6bd00b9db374bda7c7ce48e452da9eb614c526343da8bcb"} Oct 30 21:47:50.716406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.716352 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg" Oct 30 21:47:50.716520 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.716413 4921 scope.go:122] "RemoveContainer" containerID="35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd" Oct 30 21:47:50.726534 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.726519 4921 scope.go:122] "RemoveContainer" containerID="35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd" Oct 30 21:47:50.726816 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:47:50.726794 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd\": container with ID starting with 35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd not found: ID does not exist" containerID="35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd" Oct 30 21:47:50.726893 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.726830 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd"} err="failed to get container status \"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd\": rpc error: code = NotFound desc = could not find container \"35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd\": container with ID starting with 35305c9826f4d81b6c918b6adb0af4247bc33f342c77aa2329d52618d422aafd not found: ID does not exist" Oct 30 21:47:50.794389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.794308 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbk9\" (UniqueName: \"kubernetes.io/projected/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-kube-api-access-kmbk9\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:50.794490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.794452 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:50.794490 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.794473 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8e7af9-57c0-4000-8d8c-5ee30662aaef-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:50.829302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.829250 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7193-676f26" podStartSLOduration=2.829235437 podStartE2EDuration="2.829235437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.827784896 +0000 UTC m=+2184.810485314" watchObservedRunningTime="2026-10-30 21:47:50.829235437 +0000 UTC m=+2184.811935855" Oct 30 21:47:50.939262 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.939092 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:47:50.942385 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:50.942354 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f52c3-57bc69468-95ccg"] Oct 30 21:47:52.167807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:52.167772 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" path="/var/lib/kubelet/pods/fc8e7af9-57c0-4000-8d8c-5ee30662aaef/volumes" Oct 30 21:47:53.696302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.696248 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:47:53.740943 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.740912 4921 generic.go:356] "Generic (PLEG): container finished" podID="8e48e41c-69b2-452e-af41-c50416fa9501" containerID="66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407" exitCode=0 Oct 30 21:47:53.741084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.740972 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" Oct 30 21:47:53.741084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.740986 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerDied","Data":"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407"} Oct 30 21:47:53.741084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.741041 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m" event={"ID":"8e48e41c-69b2-452e-af41-c50416fa9501","Type":"ContainerDied","Data":"285093d9759bbd895251a854b9453db705f453ad7beba00df708f9de2e6f7b56"} Oct 30 21:47:53.741084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.741063 4921 scope.go:122] "RemoveContainer" containerID="66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407" Oct 30 21:47:53.750977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.750959 4921 scope.go:122] "RemoveContainer" containerID="8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e" Oct 30 21:47:53.759759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.759737 4921 scope.go:122] "RemoveContainer" containerID="66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407" Oct 30 21:47:53.760028 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:47:53.760008 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407\": container with ID starting with 66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407 not found: ID does not exist" containerID="66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407" Oct 30 21:47:53.760092 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.760036 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407"} err="failed to get container status \"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407\": rpc error: code = NotFound desc = could not find container \"66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407\": container with ID starting with 66f3d3bdd7b1b4521efb2391e86e6393780ba5f22d74eefd69f02a80744b3407 not found: ID does not exist" Oct 30 21:47:53.760092 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.760055 4921 scope.go:122] "RemoveContainer" containerID="8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e" Oct 30 21:47:53.760261 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:47:53.760244 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e\": container with ID starting with 8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e not found: ID does not exist" containerID="8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e" Oct 30 21:47:53.760300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.760265 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e"} err="failed to get container status \"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e\": rpc error: code = NotFound desc = could not find container \"8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e\": container with ID starting with 8c9010736b405531b8236bbb775312f5efdd28609bfaeb9aef18772187cb604e not found: ID does not exist" Oct 30 21:47:53.831772 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.831749 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.831877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.831821 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.831951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.831904 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832008 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.831976 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832051 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832119 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832088 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832138 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832188 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832240 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:53.832356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832241 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:53.832356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832259 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\") pod \"8e48e41c-69b2-452e-af41-c50416fa9501\" (UID: \"8e48e41c-69b2-452e-af41-c50416fa9501\") " Oct 30 21:47:53.832468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832410 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:53.832525 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832500 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:53.832686 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832664 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:53.832847 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832826 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.832931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832853 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.832931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832867 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.832931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832882 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.832931 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.832895 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e48e41c-69b2-452e-af41-c50416fa9501-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.834426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.834378 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:53.834532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.834481 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:53.834812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.834786 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:53.834911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.834885 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2" pod "8e48e41c-69b2-452e-af41-c50416fa9501" (UID: "8e48e41c-69b2-452e-af41-c50416fa9501"). InnerVolumeSpecName "kube-api-access-r9rr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:53.922383 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.922351 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:47:53.933824 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.933796 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.933939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.933828 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e48e41c-69b2-452e-af41-c50416fa9501-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.933939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.933845 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rr2\" (UniqueName: \"kubernetes.io/projected/8e48e41c-69b2-452e-af41-c50416fa9501-kube-api-access-r9rr2\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:53.933939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:53.933860 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e48e41c-69b2-452e-af41-c50416fa9501-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:54.170346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:54.170297 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:47:54.170500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:54.170359 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-99003c-openshift-default-6f574fc469-5dm8m"] Oct 30 21:47:56.171411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:47:56.171371 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" path="/var/lib/kubelet/pods/8e48e41c-69b2-452e-af41-c50416fa9501/volumes" Oct 30 21:48:01.052557 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:01.052529 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:48:06.053638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:06.053580 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" containerID="cri-o://79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b" gracePeriod=25 Oct 30 21:48:07.540374 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:07.540292 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:48:08.942381 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:08.942346 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" containerID="cri-o://1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86" gracePeriod=45 Oct 30 21:48:09.224073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.224051 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:48:09.294830 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.294804 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.294982 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.294903 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.295048 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295018 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.295111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295052 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.295111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295096 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\" (UniqueName: \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.295218 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295180 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\") pod \"2c98345b-c8d0-451f-8f01-47fa2499eb19\" (UID: \"2c98345b-c8d0-451f-8f01-47fa2499eb19\") " Oct 30 21:48:09.295290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295241 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:09.295407 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295296 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:09.295548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295528 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:09.295728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295713 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.295798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295735 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.295798 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.295750 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.297471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.297451 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:09.297569 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.297549 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "kube-api-access-tnlgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:09.297608 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.297556 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm" pod "2c98345b-c8d0-451f-8f01-47fa2499eb19" (UID: "2c98345b-c8d0-451f-8f01-47fa2499eb19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:09.397033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.397003 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c98345b-c8d0-451f-8f01-47fa2499eb19-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.397033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.397032 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnlgk\" (UniqueName: \"kubernetes.io/projected/2c98345b-c8d0-451f-8f01-47fa2499eb19-kube-api-access-tnlgk\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.397213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.397043 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c98345b-c8d0-451f-8f01-47fa2499eb19-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:09.852542 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.852508 4921 generic.go:356] "Generic (PLEG): container finished" podID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerID="1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86" exitCode=0 Oct 30 21:48:09.852742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.852548 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" Oct 30 21:48:09.852742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.852587 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" event={"ID":"2c98345b-c8d0-451f-8f01-47fa2499eb19","Type":"ContainerDied","Data":"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86"} Oct 30 21:48:09.852742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.852637 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2" event={"ID":"2c98345b-c8d0-451f-8f01-47fa2499eb19","Type":"ContainerDied","Data":"b54493122eb863a1d045946a230fde33bf5a3a43b4c0e6221e19d724ed0c688a"} Oct 30 21:48:09.852742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.852662 4921 scope.go:122] "RemoveContainer" containerID="1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86" Oct 30 21:48:09.862951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.862935 4921 scope.go:122] "RemoveContainer" containerID="1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86" Oct 30 21:48:09.863212 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:09.863195 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86\": container with ID starting with 1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86 not found: ID does not exist" containerID="1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86" Oct 30 21:48:09.863258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:09.863221 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86"} err="failed to get container status \"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86\": rpc error: code = NotFound desc = could not find container \"1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86\": container with ID starting with 1016257e383d2cbf13f8f8b3e8cbbbc0fe4bdb212e2c7073d34e95bdb8b2ec86 not found: ID does not exist" Oct 30 21:48:10.015699 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:10.015667 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:48:10.062812 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:10.062783 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-38b832/distinct-2-w2-38b832-kserve-8d76b6b96-r8hs2"] Oct 30 21:48:10.167807 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:10.167733 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" path="/var/lib/kubelet/pods/2c98345b-c8d0-451f-8f01-47fa2499eb19/volumes" Oct 30 21:48:15.874400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:15.874358 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:48:25.875202 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:25.875161 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:48:26.321973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.321931 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:48:26.471012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.470984 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\") pod \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " Oct 30 21:48:26.471169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.471157 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\" (UniqueName: \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\") pod \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\" (UID: \"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29\") " Oct 30 21:48:26.471456 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.471430 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume" pod "418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" (UID: "418cfe05-e0f9-42ff-bcc9-2aebc69e1f29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:26.471656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.471633 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.473229 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.473205 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz" pod "418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" (UID: "418cfe05-e0f9-42ff-bcc9-2aebc69e1f29"). InnerVolumeSpecName "kube-api-access-jk5bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:26.572187 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.572166 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk5bz\" (UniqueName: \"kubernetes.io/projected/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29-kube-api-access-jk5bz\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.985500 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.985471 4921 generic.go:356] "Generic (PLEG): container finished" podID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerID="79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b" exitCode=0 Oct 30 21:48:26.986006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.985511 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" event={"ID":"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29","Type":"ContainerDied","Data":"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b"} Oct 30 21:48:26.986006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.985514 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" Oct 30 21:48:26.986006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.985575 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6" event={"ID":"418cfe05-e0f9-42ff-bcc9-2aebc69e1f29","Type":"ContainerDied","Data":"c8fdae2459bfa1c75357442ca59df56247ab5cf51ed9b9879132f22d02ad862f"} Oct 30 21:48:26.986006 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.985596 4921 scope.go:122] "RemoveContainer" containerID="79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b" Oct 30 21:48:26.997496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.997473 4921 scope.go:122] "RemoveContainer" containerID="79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b" Oct 30 21:48:26.997792 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:26.997769 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b\": container with ID starting with 79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b not found: ID does not exist" containerID="79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b" Oct 30 21:48:26.997858 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:26.997803 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b"} err="failed to get container status \"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b\": rpc error: code = NotFound desc = could not find container \"79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b\": container with ID starting with 79599e8d213f5254bf99a5a3f39492b53dfba3c39be40926d201ea167f0f9b6b not found: ID does not exist" Oct 30 21:48:27.148189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:27.148162 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:48:27.197179 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:27.197155 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-38b832-5d5785fd47-4pbx6"] Oct 30 21:48:28.168067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:28.168034 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" path="/var/lib/kubelet/pods/418cfe05-e0f9-42ff-bcc9-2aebc69e1f29/volumes" Oct 30 21:48:37.539914 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:37.539879 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:48:51.715010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:51.714971 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7193-676f26"] Oct 30 21:48:51.715593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:51.715179 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7193-676f26" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" containerName="test-curl-7193-676f26" containerID="cri-o://2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84" gracePeriod=2 Oct 30 21:48:51.719936 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:51.718894 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7193-676f26"] Oct 30 21:48:51.766469 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:51.766435 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7193-676f26\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" pod="opendatahub/test-curl-7193-676f26" Oct 30 21:48:53.949707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:53.949682 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-676f26" Oct 30 21:48:54.065291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.065218 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\" (UniqueName: \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\") pod \"276d2c75-39cf-489c-abfe-f9294dbc250d\" (UID: \"276d2c75-39cf-489c-abfe-f9294dbc250d\") " Oct 30 21:48:54.067256 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.067231 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7" pod "276d2c75-39cf-489c-abfe-f9294dbc250d" (UID: "276d2c75-39cf-489c-abfe-f9294dbc250d"). InnerVolumeSpecName "kube-api-access-4qnw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:54.166768 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.166736 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qnw7\" (UniqueName: \"kubernetes.io/projected/276d2c75-39cf-489c-abfe-f9294dbc250d-kube-api-access-4qnw7\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:54.169247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.169214 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" path="/var/lib/kubelet/pods/276d2c75-39cf-489c-abfe-f9294dbc250d/volumes" Oct 30 21:48:54.197090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.197068 4921 generic.go:356] "Generic (PLEG): container finished" podID="276d2c75-39cf-489c-abfe-f9294dbc250d" containerID="2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84" exitCode=137 Oct 30 21:48:54.197185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.197111 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7193-676f26" Oct 30 21:48:54.197185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.197163 4921 scope.go:122] "RemoveContainer" containerID="2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84" Oct 30 21:48:54.206140 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.206126 4921 scope.go:122] "RemoveContainer" containerID="2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84" Oct 30 21:48:54.206418 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:48:54.206397 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84\": container with ID starting with 2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84 not found: ID does not exist" containerID="2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84" Oct 30 21:48:54.206484 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:48:54.206425 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84"} err="failed to get container status \"2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84\": rpc error: code = NotFound desc = could not find container \"2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84\": container with ID starting with 2ce83c2ecb5b8e1c7cbc86ad6e06b433f3fe9acac21caf48a0fb071a71883e84 not found: ID does not exist" Oct 30 21:49:07.540899 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:07.540863 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:49:07.942028 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:07.941954 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:49:07.942214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:07.942192 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" podUID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" containerName="maas-api" containerID="cri-o://71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87" gracePeriod=30 Oct 30 21:49:08.230124 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.230101 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:49:08.321360 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.321311 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\") pod \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " Oct 30 21:49:08.321761 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.321740 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\") pod \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " Oct 30 21:49:08.322486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.322464 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") pod \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " Oct 30 21:49:08.322720 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.322701 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\" (UniqueName: \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\") pod \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\" (UID: \"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33\") " Oct 30 21:49:08.324612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.324589 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca" pod "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" (UID: "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:08.326010 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.325842 4921 generic.go:356] "Generic (PLEG): container finished" podID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" containerID="71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87" exitCode=0 Oct 30 21:49:08.326115 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.325951 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" Oct 30 21:49:08.326174 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.325970 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" event={"ID":"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33","Type":"ContainerDied","Data":"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87"} Oct 30 21:49:08.326233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.326189 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw" event={"ID":"9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33","Type":"ContainerDied","Data":"7a46cb96a57bc2fe72c079e17c35ef057e291968d7c3a63a4b07edfa0d80c664"} Oct 30 21:49:08.326295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.326229 4921 scope.go:122] "RemoveContainer" containerID="71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87" Oct 30 21:49:08.331212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.331094 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs" pod "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" (UID: "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:08.331212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.331188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls" pod "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" (UID: "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:08.334743 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.334713 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh" pod "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" (UID: "9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33"). InnerVolumeSpecName "kube-api-access-lzvzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:08.372767 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.372733 4921 scope.go:122] "RemoveContainer" containerID="71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87" Oct 30 21:49:08.373128 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:08.373109 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87\": container with ID starting with 71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87 not found: ID does not exist" containerID="71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87" Oct 30 21:49:08.373188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.373140 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87"} err="failed to get container status \"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87\": rpc error: code = NotFound desc = could not find container \"71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87\": container with ID starting with 71e1164699d677bc3b5f5c7901e417c71d5eff513a56858761107f5ad81fbb87 not found: ID does not exist" Oct 30 21:49:08.424355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.424298 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.424355 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.424356 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.424578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.424366 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.424578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.424376 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvzh\" (UniqueName: \"kubernetes.io/projected/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33-kube-api-access-lzvzh\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:08.429436 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429410 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7196-7b92f8"] Oct 30 21:49:08.429911 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429899 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" containerName="maas-api" Oct 30 21:49:08.429951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429913 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" containerName="maas-api" Oct 30 21:49:08.429951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429931 4921 cpu_manager.go:414] "containerMap: removing container" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" containerName="test-curl-7193-676f26" Oct 30 21:49:08.429951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429936 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" containerName="test-curl-7193-676f26" Oct 30 21:49:08.429951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429943 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:49:08.429951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429948 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429973 4921 cpu_manager.go:414] "containerMap: removing container" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerName="maas-api" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429978 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerName="maas-api" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429987 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.429992 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430000 4921 cpu_manager.go:414] "containerMap: removing container" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430005 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430072 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8e7af9-57c0-4000-8d8c-5ee30662aaef" containerName="maas-api" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430080 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c98345b-c8d0-451f-8f01-47fa2499eb19" containerName="main" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430086 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="418cfe05-e0f9-42ff-bcc9-2aebc69e1f29" containerName="payload-pre-processing" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430092 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430099 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" containerName="maas-api" Oct 30 21:49:08.430106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430106 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="276d2c75-39cf-489c-abfe-f9294dbc250d" containerName="test-curl-7193-676f26" Oct 30 21:49:08.430509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.430112 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:49:08.434818 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.434800 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:08.489920 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.489834 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-7b92f8"] Oct 30 21:49:08.512964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.512934 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pmshp\"" Oct 30 21:49:08.627094 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.627061 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq69w\" (UniqueName: \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\") pod \"test-curl-7196-7b92f8\" (UID: \"62285106-36ce-4dc8-9ca6-e7f83774fb23\") " pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:08.728351 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.728297 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq69w\" (UniqueName: \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\") pod \"test-curl-7196-7b92f8\" (UID: \"62285106-36ce-4dc8-9ca6-e7f83774fb23\") " pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:08.746692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.746615 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:49:08.749723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.749703 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-38b832-d459db5d7-bqdlw"] Oct 30 21:49:08.784662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:08.784643 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq69w\" (UniqueName: \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\") pod \"test-curl-7196-7b92f8\" (UID: \"62285106-36ce-4dc8-9ca6-e7f83774fb23\") " pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:09.047053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:09.046971 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:09.417158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:09.417129 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7196-7b92f8"] Oct 30 21:49:09.418043 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:49:09.418014 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62285106_36ce_4dc8_9ca6_e7f83774fb23.slice/crio-fafac90950414c64b7c76f6c76d61d04c0fc60ee28c497b14195448d443be46c WatchSource:0}: Error finding container fafac90950414c64b7c76f6c76d61d04c0fc60ee28c497b14195448d443be46c: Status 404 returned error can't find the container with id fafac90950414c64b7c76f6c76d61d04c0fc60ee28c497b14195448d443be46c Oct 30 21:49:10.170121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:10.170081 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33" path="/var/lib/kubelet/pods/9b08a3fc-adb2-4c90-9f4e-d0cd91e5ee33/volumes" Oct 30 21:49:10.341621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:10.341583 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-7b92f8" event={"ID":"62285106-36ce-4dc8-9ca6-e7f83774fb23","Type":"ContainerStarted","Data":"2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4"} Oct 30 21:49:10.341621 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:10.341621 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7196-7b92f8" event={"ID":"62285106-36ce-4dc8-9ca6-e7f83774fb23","Type":"ContainerStarted","Data":"fafac90950414c64b7c76f6c76d61d04c0fc60ee28c497b14195448d443be46c"} Oct 30 21:49:10.468559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:10.468511 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7196-7b92f8" podStartSLOduration=2.468497504 podStartE2EDuration="2.468497504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.466930944 +0000 UTC m=+2264.449631362" watchObservedRunningTime="2026-10-30 21:49:10.468497504 +0000 UTC m=+2264.451197921" Oct 30 21:49:11.885918 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:11.885886 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7196-7b92f8"] Oct 30 21:49:11.889977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:11.889947 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7196-7b92f8"] Oct 30 21:49:12.353594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:12.353552 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7196-7b92f8" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" containerName="test-curl-7196-7b92f8" containerID="cri-o://2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4" gracePeriod=2 Oct 30 21:49:14.593592 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:14.593565 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:14.726961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:14.726937 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\" (UniqueName: \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\") pod \"62285106-36ce-4dc8-9ca6-e7f83774fb23\" (UID: \"62285106-36ce-4dc8-9ca6-e7f83774fb23\") " Oct 30 21:49:14.728949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:14.728922 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w" pod "62285106-36ce-4dc8-9ca6-e7f83774fb23" (UID: "62285106-36ce-4dc8-9ca6-e7f83774fb23"). InnerVolumeSpecName "kube-api-access-mq69w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:14.827988 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:14.827964 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq69w\" (UniqueName: \"kubernetes.io/projected/62285106-36ce-4dc8-9ca6-e7f83774fb23-kube-api-access-mq69w\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:15.388346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:15.382822 4921 generic.go:356] "Generic (PLEG): container finished" podID="62285106-36ce-4dc8-9ca6-e7f83774fb23" containerID="2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4" exitCode=137 Oct 30 21:49:15.388346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:15.382883 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:15.388346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:15.382987 4921 scope.go:122] "RemoveContainer" containerID="2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4" Oct 30 21:49:15.394100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:15.394080 4921 scope.go:122] "RemoveContainer" containerID="2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4" Oct 30 21:49:15.394376 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:15.394356 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4\": container with ID starting with 2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4 not found: ID does not exist" containerID="2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4" Oct 30 21:49:15.394451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:15.394387 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4"} err="failed to get container status \"2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4\": rpc error: code = NotFound desc = could not find container \"2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4\": container with ID starting with 2fb764174e8d2b7bc0d9674bc19ad8cb1505415fbf951c6877fff8bd731748d4 not found: ID does not exist" Oct 30 21:49:15.461586 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:15.461539 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-7b92f8\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:16.167640 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:16.167608 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" path="/var/lib/kubelet/pods/62285106-36ce-4dc8-9ca6-e7f83774fb23/volumes" Oct 30 21:49:16.241606 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:16.241564 4921 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7196-7b92f8\" is forbidden: User \"system:node:ip-10-0-137-220.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-220.us-west-1.compute.internal' and this object" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" pod="opendatahub/test-curl-7196-7b92f8" Oct 30 21:49:18.008284 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:18.008250 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:49:18.008703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:18.008514 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" containerID="cri-o://47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812" gracePeriod=30 Oct 30 21:49:18.022511 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:18.022480 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:49:18.022721 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:18.022687 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" containerID="cri-o://ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f" gracePeriod=30 Oct 30 21:49:22.494983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:22.494946 4921 patch_prober.go:29] interesting pod/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:22.495412 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:22.495021 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:23.280305 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.280249 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:49:23.312587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.312567 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:49:23.425829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.425801 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.425948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.425890 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.425948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.425921 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.425963 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426003 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426021 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426061 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426071 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426110 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426142 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426192 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426220 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426250 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426280 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426329 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426307 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426360 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426386 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\") pod \"c10258cf-0d7a-47f2-a2df-4276659beb18\" (UID: \"c10258cf-0d7a-47f2-a2df-4276659beb18\") " Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426405 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426453 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\") pod \"2257752d-2b24-4829-af38-5e4e8f02f3b9\" (UID: \"2257752d-2b24-4829-af38-5e4e8f02f3b9\") " Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426488 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.426759 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426636 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.427096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426953 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.427096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426978 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.427096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.426994 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.427265 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.427487 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.427787 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.428060 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.428078 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.428140 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.428548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.428483 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:23.429976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.429948 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:23.430958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.430906 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.430958 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.430922 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "kube-api-access-bz9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:23.431122 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.430923 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:23.431430 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.431405 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:23.432379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.432112 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:23.432474 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.432393 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy" pod "2257752d-2b24-4829-af38-5e4e8f02f3b9" (UID: "2257752d-2b24-4829-af38-5e4e8f02f3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:23.432644 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.432617 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q" pod "c10258cf-0d7a-47f2-a2df-4276659beb18" (UID: "c10258cf-0d7a-47f2-a2df-4276659beb18"). InnerVolumeSpecName "kube-api-access-pcn8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:23.440193 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.440165 4921 generic.go:356] "Generic (PLEG): container finished" podID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerID="ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f" exitCode=0 Oct 30 21:49:23.440343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.440286 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" event={"ID":"2257752d-2b24-4829-af38-5e4e8f02f3b9","Type":"ContainerDied","Data":"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f"} Oct 30 21:49:23.440343 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.440338 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" event={"ID":"2257752d-2b24-4829-af38-5e4e8f02f3b9","Type":"ContainerDied","Data":"1c26865b33dede4445e7c0dc88f401e91e56fe957be27de02cc32b0fc895512f"} Oct 30 21:49:23.440473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.440364 4921 scope.go:122] "RemoveContainer" containerID="ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f" Oct 30 21:49:23.440473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.440459 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7" Oct 30 21:49:23.445447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.445286 4921 generic.go:356] "Generic (PLEG): container finished" podID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerID="47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812" exitCode=0 Oct 30 21:49:23.445447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.445385 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" event={"ID":"c10258cf-0d7a-47f2-a2df-4276659beb18","Type":"ContainerDied","Data":"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812"} Oct 30 21:49:23.445447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.445425 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" event={"ID":"c10258cf-0d7a-47f2-a2df-4276659beb18","Type":"ContainerDied","Data":"7af0f062f9ea3ee241343a77ef27b6f8d94ea313515a6d8a2147b13e035aea1b"} Oct 30 21:49:23.445447 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.445432 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w" Oct 30 21:49:23.458116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.458095 4921 scope.go:122] "RemoveContainer" containerID="ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f" Oct 30 21:49:23.458448 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:23.458418 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f\": container with ID starting with ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f not found: ID does not exist" containerID="ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f" Oct 30 21:49:23.458551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.458458 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f"} err="failed to get container status \"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f\": rpc error: code = NotFound desc = could not find container \"ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f\": container with ID starting with ef2408363337b974b85dcef2656eb51e65336aea3f002ba93b6e2305ccb8fa8f not found: ID does not exist" Oct 30 21:49:23.458551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.458481 4921 scope.go:122] "RemoveContainer" containerID="47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812" Oct 30 21:49:23.469981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.469955 4921 scope.go:122] "RemoveContainer" containerID="47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812" Oct 30 21:49:23.470263 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:23.470242 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812\": container with ID starting with 47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812 not found: ID does not exist" containerID="47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812" Oct 30 21:49:23.470342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.470272 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812"} err="failed to get container status \"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812\": rpc error: code = NotFound desc = could not find container \"47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812\": container with ID starting with 47d288164a42edf94232b786ac365ad28925acaa54fc95bd811bef229eab5812 not found: ID does not exist" Oct 30 21:49:23.528507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528475 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.528507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528501 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.528507 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528510 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528520 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528528 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528537 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528545 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528553 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2257752d-2b24-4829-af38-5e4e8f02f3b9-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528561 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c10258cf-0d7a-47f2-a2df-4276659beb18-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528569 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528576 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528584 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c10258cf-0d7a-47f2-a2df-4276659beb18-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528593 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcn8q\" (UniqueName: \"kubernetes.io/projected/c10258cf-0d7a-47f2-a2df-4276659beb18-kube-api-access-pcn8q\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528601 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2257752d-2b24-4829-af38-5e4e8f02f3b9-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.529080 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.528610 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9cl\" (UniqueName: \"kubernetes.io/projected/2257752d-2b24-4829-af38-5e4e8f02f3b9-kube-api-access-bz9cl\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:23.731949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.731909 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:49:23.734954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.734928 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-lbn6w"] Oct 30 21:49:23.778114 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.778091 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:49:23.778362 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.778265 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" podUID="0beb01b5-0304-47d4-b704-e6510860a6b8" containerName="maas-api" containerID="cri-o://d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde" gracePeriod=30 Oct 30 21:49:23.845020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.844991 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:49:23.847961 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:23.847935 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-38b832-openshift-default-849f7b9d8f-29nq7"] Oct 30 21:49:24.063868 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.063840 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:49:24.168765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.168723 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" path="/var/lib/kubelet/pods/2257752d-2b24-4829-af38-5e4e8f02f3b9/volumes" Oct 30 21:49:24.169379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.169356 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" path="/var/lib/kubelet/pods/c10258cf-0d7a-47f2-a2df-4276659beb18/volumes" Oct 30 21:49:24.236620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.236587 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\" (UniqueName: \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\") pod \"0beb01b5-0304-47d4-b704-e6510860a6b8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " Oct 30 21:49:24.236822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.236650 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") pod \"0beb01b5-0304-47d4-b704-e6510860a6b8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " Oct 30 21:49:24.236822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.236720 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\") pod \"0beb01b5-0304-47d4-b704-e6510860a6b8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " Oct 30 21:49:24.236959 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.236881 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\") pod \"0beb01b5-0304-47d4-b704-e6510860a6b8\" (UID: \"0beb01b5-0304-47d4-b704-e6510860a6b8\") " Oct 30 21:49:24.237101 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.237066 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca" pod "0beb01b5-0304-47d4-b704-e6510860a6b8" (UID: "0beb01b5-0304-47d4-b704-e6510860a6b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:24.237391 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.237370 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0beb01b5-0304-47d4-b704-e6510860a6b8-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:24.239338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.239303 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs" pod "0beb01b5-0304-47d4-b704-e6510860a6b8" (UID: "0beb01b5-0304-47d4-b704-e6510860a6b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:24.239452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.239362 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls" pod "0beb01b5-0304-47d4-b704-e6510860a6b8" (UID: "0beb01b5-0304-47d4-b704-e6510860a6b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:24.239452 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.239393 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75" pod "0beb01b5-0304-47d4-b704-e6510860a6b8" (UID: "0beb01b5-0304-47d4-b704-e6510860a6b8"). InnerVolumeSpecName "kube-api-access-s7m75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:24.338890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.338836 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:24.338890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.338860 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7m75\" (UniqueName: \"kubernetes.io/projected/0beb01b5-0304-47d4-b704-e6510860a6b8-kube-api-access-s7m75\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:24.338890 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.338870 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0beb01b5-0304-47d4-b704-e6510860a6b8-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:24.454707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.454671 4921 generic.go:356] "Generic (PLEG): container finished" podID="0beb01b5-0304-47d4-b704-e6510860a6b8" containerID="d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde" exitCode=0 Oct 30 21:49:24.454874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.454719 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" Oct 30 21:49:24.454874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.454747 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" event={"ID":"0beb01b5-0304-47d4-b704-e6510860a6b8","Type":"ContainerDied","Data":"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde"} Oct 30 21:49:24.454874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.454801 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8" event={"ID":"0beb01b5-0304-47d4-b704-e6510860a6b8","Type":"ContainerDied","Data":"1758ee1e2994a71246d0525266fec2c757200ce184664ee445c2ffb86d59f768"} Oct 30 21:49:24.454874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.454827 4921 scope.go:122] "RemoveContainer" containerID="d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde" Oct 30 21:49:24.465948 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.465932 4921 scope.go:122] "RemoveContainer" containerID="d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde" Oct 30 21:49:24.466209 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:24.466192 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde\": container with ID starting with d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde not found: ID does not exist" containerID="d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde" Oct 30 21:49:24.466274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.466222 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde"} err="failed to get container status \"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde\": rpc error: code = NotFound desc = could not find container \"d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde\": container with ID starting with d94b2bd3e388e50b523fd4141d716da2b2fb719e17297c5db043c95b1e34fdde not found: ID does not exist" Oct 30 21:49:24.617748 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.617678 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:49:24.621061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:24.621034 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fe5385-657b69fd94-7k4k8"] Oct 30 21:49:26.168527 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:26.168491 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0beb01b5-0304-47d4-b704-e6510860a6b8" path="/var/lib/kubelet/pods/0beb01b5-0304-47d4-b704-e6510860a6b8/volumes" Oct 30 21:49:37.540256 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:37.540178 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:49:58.804712 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:58.804680 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:49:58.805310 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:58.804913 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" podUID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" containerName="maas-api" containerID="cri-o://4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190" gracePeriod=30 Oct 30 21:49:59.073037 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.073015 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:49:59.220035 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.219993 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\") pod \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " Oct 30 21:49:59.220212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.220125 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\") pod \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " Oct 30 21:49:59.220212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.220188 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\" (UniqueName: \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\") pod \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " Oct 30 21:49:59.220295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.220259 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") pod \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\" (UID: \"880b8389-ecbc-4dbc-99ae-9af514cda5ae\") " Oct 30 21:49:59.220587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.220560 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca" pod "880b8389-ecbc-4dbc-99ae-9af514cda5ae" (UID: "880b8389-ecbc-4dbc-99ae-9af514cda5ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:59.220848 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.220772 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/880b8389-ecbc-4dbc-99ae-9af514cda5ae-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.222461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.222438 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls" pod "880b8389-ecbc-4dbc-99ae-9af514cda5ae" (UID: "880b8389-ecbc-4dbc-99ae-9af514cda5ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:59.222671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.222650 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs" pod "880b8389-ecbc-4dbc-99ae-9af514cda5ae" (UID: "880b8389-ecbc-4dbc-99ae-9af514cda5ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:59.222671 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.222667 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx" pod "880b8389-ecbc-4dbc-99ae-9af514cda5ae" (UID: "880b8389-ecbc-4dbc-99ae-9af514cda5ae"). InnerVolumeSpecName "kube-api-access-jt4sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:59.322170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.322084 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt4sx\" (UniqueName: \"kubernetes.io/projected/880b8389-ecbc-4dbc-99ae-9af514cda5ae-kube-api-access-jt4sx\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.322170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.322112 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.322170 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.322121 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/880b8389-ecbc-4dbc-99ae-9af514cda5ae-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.694649 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.694572 4921 generic.go:356] "Generic (PLEG): container finished" podID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" containerID="4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190" exitCode=0 Oct 30 21:49:59.694649 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.694619 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" Oct 30 21:49:59.694822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.694653 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" event={"ID":"880b8389-ecbc-4dbc-99ae-9af514cda5ae","Type":"ContainerDied","Data":"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190"} Oct 30 21:49:59.694822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.694707 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps" event={"ID":"880b8389-ecbc-4dbc-99ae-9af514cda5ae","Type":"ContainerDied","Data":"091a2b6c98cb3029f995ca88ea6f550e2dde1e7669c1fa695f63c5ba954b51c8"} Oct 30 21:49:59.694822 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.694730 4921 scope.go:122] "RemoveContainer" containerID="4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190" Oct 30 21:49:59.705106 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.705087 4921 scope.go:122] "RemoveContainer" containerID="4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190" Oct 30 21:49:59.705416 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:49:59.705394 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190\": container with ID starting with 4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190 not found: ID does not exist" containerID="4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190" Oct 30 21:49:59.705531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.705424 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190"} err="failed to get container status \"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190\": rpc error: code = NotFound desc = could not find container \"4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190\": container with ID starting with 4e5054c32c5c4da887fa5af4d06cc38e0e7633309174140ac4e98bb245a93190 not found: ID does not exist" Oct 30 21:49:59.860255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.860220 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:49:59.909585 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:49:59.909554 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9085f4-6dd794bc7c-7bzps"] Oct 30 21:50:00.168097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:00.168055 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" path="/var/lib/kubelet/pods/880b8389-ecbc-4dbc-99ae-9af514cda5ae/volumes" Oct 30 21:50:07.541539 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:07.541496 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:50:08.503705 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:08.503666 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:50:08.503932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:08.503893 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" containerID="cri-o://3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be" gracePeriod=30 Oct 30 21:50:10.075016 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:10.074984 4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:50:10.075411 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:10.075028 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:50:13.765574 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.765518 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:50:13.793053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.793023 4921 generic.go:356] "Generic (PLEG): container finished" podID="858ec943-5d7f-429f-8de2-ef330e71578a" containerID="3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be" exitCode=0 Oct 30 21:50:13.793172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.793058 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" Oct 30 21:50:13.793172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.793102 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerDied","Data":"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be"} Oct 30 21:50:13.793258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.793182 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g" event={"ID":"858ec943-5d7f-429f-8de2-ef330e71578a","Type":"ContainerDied","Data":"c4a9b7e0479b16f9e4a068c5df55de815ff814af4972398068641f7cedb392b1"} Oct 30 21:50:13.793258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.793214 4921 scope.go:122] "RemoveContainer" containerID="3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be" Oct 30 21:50:13.805654 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.805636 4921 scope.go:122] "RemoveContainer" containerID="7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e" Oct 30 21:50:13.814183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.814160 4921 scope.go:122] "RemoveContainer" containerID="3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be" Oct 30 21:50:13.814467 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:13.814441 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be\": container with ID starting with 3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be not found: ID does not exist" containerID="3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be" Oct 30 21:50:13.814518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.814478 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be"} err="failed to get container status \"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be\": rpc error: code = NotFound desc = could not find container \"3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be\": container with ID starting with 3764de0f7d3bc968b29ea046d8f495a0aa22025bbea7438701ae3f7143cfc0be not found: ID does not exist" Oct 30 21:50:13.814518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.814502 4921 scope.go:122] "RemoveContainer" containerID="7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e" Oct 30 21:50:13.814766 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:13.814748 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e\": container with ID starting with 7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e not found: ID does not exist" containerID="7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e" Oct 30 21:50:13.814810 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.814774 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e"} err="failed to get container status \"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e\": rpc error: code = NotFound desc = could not find container \"7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e\": container with ID starting with 7e26e6edd7662158697c3f598663e004ebf18c61cb8b08302018d8f6e847b52e not found: ID does not exist" Oct 30 21:50:13.881364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881340 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881376 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881400 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881441 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881465 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881461 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881562 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881606 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881637 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.881685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881668 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.881908 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881740 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\") pod \"858ec943-5d7f-429f-8de2-ef330e71578a\" (UID: \"858ec943-5d7f-429f-8de2-ef330e71578a\") " Oct 30 21:50:13.882189 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.882168 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.882344 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881760 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.882486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.881872 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.882486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.882121 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:13.882486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.882188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.883659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.883636 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:13.883769 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.883692 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.883916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.883897 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "kube-api-access-2bhxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:13.883979 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.883957 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo" pod "858ec943-5d7f-429f-8de2-ef330e71578a" (UID: "858ec943-5d7f-429f-8de2-ef330e71578a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:13.928183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.928163 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:50:13.983198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983168 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858ec943-5d7f-429f-8de2-ef330e71578a-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983201 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983218 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983232 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858ec943-5d7f-429f-8de2-ef330e71578a-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983244 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhxg\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-kube-api-access-2bhxg\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983258 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983273 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858ec943-5d7f-429f-8de2-ef330e71578a-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:13.983354 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:13.983285 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858ec943-5d7f-429f-8de2-ef330e71578a-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:14.215482 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:14.215425 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:50:14.218916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:14.218881 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9085f4-openshift-default-dcc696b85-pmf8g"] Oct 30 21:50:14.869852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:14.869815 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:50:14.870286 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:14.870019 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" podUID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" containerName="maas-api" containerID="cri-o://e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3" gracePeriod=30 Oct 30 21:50:15.143230 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.143204 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:50:15.199893 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.199856 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\") pod \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " Oct 30 21:50:15.200032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.199994 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") pod \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " Oct 30 21:50:15.200097 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.200079 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\" (UniqueName: \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\") pod \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " Oct 30 21:50:15.200214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.200188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca" pod "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" (UID: "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:15.200289 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.200201 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\") pod \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\" (UID: \"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149\") " Oct 30 21:50:15.200678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.200653 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:15.202174 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.202149 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls" pod "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" (UID: "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:15.202296 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.202274 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs" pod "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" (UID: "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:15.202377 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.202358 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28" pod "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" (UID: "08d45e4b-7dcc-4312-9df0-f3f7b4ae6149"). InnerVolumeSpecName "kube-api-access-5hr28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:15.302252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.302220 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:15.302252 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.302251 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr28\" (UniqueName: \"kubernetes.io/projected/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-kube-api-access-5hr28\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:15.302469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.302262 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:15.818375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.818336 4921 generic.go:356] "Generic (PLEG): container finished" podID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" containerID="e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3" exitCode=0 Oct 30 21:50:15.818375 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.818381 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" Oct 30 21:50:15.818591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.818421 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" event={"ID":"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149","Type":"ContainerDied","Data":"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3"} Oct 30 21:50:15.818591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.818477 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq" event={"ID":"08d45e4b-7dcc-4312-9df0-f3f7b4ae6149","Type":"ContainerDied","Data":"6ac86252ca438f1a1ef572efa09076ee1d8a3db901a99dd4510d9319f208f95d"} Oct 30 21:50:15.818591 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.818505 4921 scope.go:122] "RemoveContainer" containerID="e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3" Oct 30 21:50:15.829635 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.829618 4921 scope.go:122] "RemoveContainer" containerID="e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3" Oct 30 21:50:15.829913 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:15.829894 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3\": container with ID starting with e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3 not found: ID does not exist" containerID="e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3" Oct 30 21:50:15.829964 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.829923 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3"} err="failed to get container status \"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3\": rpc error: code = NotFound desc = could not find container \"e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3\": container with ID starting with e010990fefe78b03028cab81eb13ed811db08e2d8c6958399173253fe66e7ef3 not found: ID does not exist" Oct 30 21:50:15.982067 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:15.982035 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:50:16.030505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:16.030472 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-hbwdq"] Oct 30 21:50:16.167976 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:16.167902 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" path="/var/lib/kubelet/pods/08d45e4b-7dcc-4312-9df0-f3f7b4ae6149/volumes" Oct 30 21:50:16.168407 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:16.168393 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" path="/var/lib/kubelet/pods/858ec943-5d7f-429f-8de2-ef330e71578a/volumes" Oct 30 21:50:26.291394 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291364 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq"] Oct 30 21:50:26.291808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291795 4921 cpu_manager.go:414] "containerMap: removing container" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291808 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291818 4921 cpu_manager.go:414] "containerMap: removing container" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" containerName="test-curl-7196-7b92f8" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291824 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" containerName="test-curl-7196-7b92f8" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291834 4921 cpu_manager.go:414] "containerMap: removing container" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291838 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" Oct 30 21:50:26.291856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291854 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291859 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e48e41c-69b2-452e-af41-c50416fa9501" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291869 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291874 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291888 4921 cpu_manager.go:414] "containerMap: removing container" podUID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291892 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291902 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0beb01b5-0304-47d4-b704-e6510860a6b8" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291907 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0beb01b5-0304-47d4-b704-e6510860a6b8" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291916 4921 cpu_manager.go:414] "containerMap: removing container" podUID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.291921 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292005 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="880b8389-ecbc-4dbc-99ae-9af514cda5ae" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292014 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0beb01b5-0304-47d4-b704-e6510860a6b8" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292021 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d45e4b-7dcc-4312-9df0-f3f7b4ae6149" containerName="maas-api" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292031 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292037 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="62285106-36ce-4dc8-9ca6-e7f83774fb23" containerName="test-curl-7196-7b92f8" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292043 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2257752d-2b24-4829-af38-5e4e8f02f3b9" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292049 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:26.292049 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.292056 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10258cf-0d7a-47f2-a2df-4276659beb18" containerName="istio-proxy" Oct 30 21:50:26.295260 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.295243 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.354519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.354490 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq"] Oct 30 21:50:26.400291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.400269 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:26.400438 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.400266 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:50:26.430306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.430274 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.430446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.430417 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.430560 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.430542 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dml\" (UniqueName: \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.531352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.531306 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dml\" (UniqueName: \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.531523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.531370 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.531523 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.531424 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.532034 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.532006 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.533742 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.533721 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.587366 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.587287 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dml\" (UniqueName: \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\") pod \"maas-api-revoke-sub-e9cd8a4a-sbmjq\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.608198 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.608180 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:26.983877 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:26.983852 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq"] Oct 30 21:50:26.985116 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:50:26.985085 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff33b735_4cad_413e_9225_95cb38cfc404.slice/crio-9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be WatchSource:0}: Error finding container 9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be: Status 404 returned error can't find the container with id 9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be Oct 30 21:50:27.395231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.393516 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:50:27.395231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.394311 4921 cpu_manager.go:414] "containerMap: removing container" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:27.395231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.394350 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858ec943-5d7f-429f-8de2-ef330e71578a" containerName="istio-proxy" Oct 30 21:50:27.398259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.398239 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.444221 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.444195 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdpqz\" (UniqueName: \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.444408 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.444254 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.450666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.450640 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3ab88248\"" Oct 30 21:50:27.450839 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.450742 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3ab88248-dockercfg-8xwtv\"" Oct 30 21:50:27.454860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.454821 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:50:27.545491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.545462 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.545662 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.545640 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdpqz\" (UniqueName: \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.546219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.546199 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.556909 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.556885 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:50:27.561972 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.561934 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.602130 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.602107 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdpqz\" (UniqueName: \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\") pod \"payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.617973 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.617945 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:50:27.646479 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.646450 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2m2w\" (UniqueName: \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.646631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.646584 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.712120 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.712096 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:27.747870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.747777 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.748001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.747931 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2m2w\" (UniqueName: \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.748536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.748514 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.810330 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.807529 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2m2w\" (UniqueName: \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\") pod \"payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.878504 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.878435 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:27.953578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.953532 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" event={"ID":"ff33b735-4cad-413e-9225-95cb38cfc404","Type":"ContainerStarted","Data":"7e27ab0d1c4f5171c644435eb2c9bf78c5a320e16f5202de4455f6e472b938d6"} Oct 30 21:50:27.953717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:27.953599 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" event={"ID":"ff33b735-4cad-413e-9225-95cb38cfc404","Type":"ContainerStarted","Data":"9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be"} Oct 30 21:50:28.070188 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.070119 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" podStartSLOduration=2.070104902 podStartE2EDuration="2.070104902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:28.068160358 +0000 UTC m=+2342.050860777" watchObservedRunningTime="2026-10-30 21:50:28.070104902 +0000 UTC m=+2342.052805319" Oct 30 21:50:28.089596 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.089568 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:50:28.091087 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:50:28.091063 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca84a6d_8e36_4939_8001_aabd34323020.slice/crio-cfdf062e9bc1a2146dd26e4dddc848e783b4b1c7e16a7933e31d68e1db2ca93a WatchSource:0}: Error finding container cfdf062e9bc1a2146dd26e4dddc848e783b4b1c7e16a7933e31d68e1db2ca93a: Status 404 returned error can't find the container with id cfdf062e9bc1a2146dd26e4dddc848e783b4b1c7e16a7933e31d68e1db2ca93a Oct 30 21:50:28.253184 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.253155 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:50:28.253984 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:50:28.253959 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a060e7_4256_482e_ad22_607dfe1acbca.slice/crio-60da011c6583cf30d2e85c178a7b387bf7d5e849788c7feb753fcb5e6f4224c6 WatchSource:0}: Error finding container 60da011c6583cf30d2e85c178a7b387bf7d5e849788c7feb753fcb5e6f4224c6: Status 404 returned error can't find the container with id 60da011c6583cf30d2e85c178a7b387bf7d5e849788c7feb753fcb5e6f4224c6 Oct 30 21:50:28.947387 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.947302 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" containerID="cri-o://1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf" gracePeriod=45 Oct 30 21:50:28.962901 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.962865 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" event={"ID":"3ca84a6d-8e36-4939-8001-aabd34323020","Type":"ContainerStarted","Data":"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948"} Oct 30 21:50:28.963024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.962931 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" event={"ID":"3ca84a6d-8e36-4939-8001-aabd34323020","Type":"ContainerStarted","Data":"cfdf062e9bc1a2146dd26e4dddc848e783b4b1c7e16a7933e31d68e1db2ca93a"} Oct 30 21:50:28.963024 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.962959 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:28.964518 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.964494 4921 generic.go:356] "Generic (PLEG): container finished" podID="ff33b735-4cad-413e-9225-95cb38cfc404" containerID="7e27ab0d1c4f5171c644435eb2c9bf78c5a320e16f5202de4455f6e472b938d6" exitCode=0 Oct 30 21:50:28.964609 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.964570 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" event={"ID":"ff33b735-4cad-413e-9225-95cb38cfc404","Type":"ContainerDied","Data":"7e27ab0d1c4f5171c644435eb2c9bf78c5a320e16f5202de4455f6e472b938d6"} Oct 30 21:50:28.966150 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.966126 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" event={"ID":"26a060e7-4256-482e-ad22-607dfe1acbca","Type":"ContainerStarted","Data":"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d"} Oct 30 21:50:28.966239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.966175 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:28.966239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:28.966192 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" event={"ID":"26a060e7-4256-482e-ad22-607dfe1acbca","Type":"ContainerStarted","Data":"60da011c6583cf30d2e85c178a7b387bf7d5e849788c7feb753fcb5e6f4224c6"} Oct 30 21:50:29.056580 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.056542 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused" Oct 30 21:50:29.082769 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.082716 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" podStartSLOduration=2.082699285 podStartE2EDuration="2.082699285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:29.077701051 +0000 UTC m=+2343.060401469" watchObservedRunningTime="2026-10-30 21:50:29.082699285 +0000 UTC m=+2343.065399701" Oct 30 21:50:29.212692 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.212603 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:50:29.263273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263213 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263310 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263384 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263461 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263512 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:29.263715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263530 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\" (UniqueName: \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263579 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\") pod \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\" (UID: \"3eb422e0-1506-4ccb-84d1-e29858c3b9cd\") " Oct 30 21:50:29.263715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263653 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:29.263932 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.263874 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:29.264077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.264022 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.264077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.264044 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.264077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.264058 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.265547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.265526 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:29.265547 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.265539 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:29.265861 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.265838 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc" pod "3eb422e0-1506-4ccb-84d1-e29858c3b9cd" (UID: "3eb422e0-1506-4ccb-84d1-e29858c3b9cd"). InnerVolumeSpecName "kube-api-access-bjnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:29.304264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.304201 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" podStartSLOduration=2.304188255 podStartE2EDuration="2.304188255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:29.303893334 +0000 UTC m=+2343.286593751" watchObservedRunningTime="2026-10-30 21:50:29.304188255 +0000 UTC m=+2343.286888673" Oct 30 21:50:29.364855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.364832 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.364855 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.364855 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.364981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.364867 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnpc\" (UniqueName: \"kubernetes.io/projected/3eb422e0-1506-4ccb-84d1-e29858c3b9cd-kube-api-access-bjnpc\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:29.975707 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.975670 4921 generic.go:356] "Generic (PLEG): container finished" podID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerID="1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf" exitCode=0 Oct 30 21:50:29.976261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.975714 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" Oct 30 21:50:29.976261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.975760 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" event={"ID":"3eb422e0-1506-4ccb-84d1-e29858c3b9cd","Type":"ContainerDied","Data":"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf"} Oct 30 21:50:29.976261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.975808 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh" event={"ID":"3eb422e0-1506-4ccb-84d1-e29858c3b9cd","Type":"ContainerDied","Data":"51eeeb404d52b97fe3f75faf52f5f1c68f951289fdeff31e622e330b02cc849e"} Oct 30 21:50:29.976261 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.975831 4921 scope.go:122] "RemoveContainer" containerID="1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf" Oct 30 21:50:29.987364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.987344 4921 scope.go:122] "RemoveContainer" containerID="1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf" Oct 30 21:50:29.987645 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:29.987627 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf\": container with ID starting with 1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf not found: ID does not exist" containerID="1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf" Oct 30 21:50:29.987728 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:29.987656 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf"} err="failed to get container status \"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf\": rpc error: code = NotFound desc = could not find container \"1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf\": container with ID starting with 1daed914387a53e2a9981e010199d60a48a22cd88b792d3aa43ec0337bb2e1cf not found: ID does not exist" Oct 30 21:50:30.122387 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.122357 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:30.138601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.138573 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:50:30.173732 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.173705 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\") pod \"ff33b735-4cad-413e-9225-95cb38cfc404\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " Oct 30 21:50:30.173870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.173753 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\" (UniqueName: \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\") pod \"ff33b735-4cad-413e-9225-95cb38cfc404\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " Oct 30 21:50:30.173870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.173853 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\") pod \"ff33b735-4cad-413e-9225-95cb38cfc404\" (UID: \"ff33b735-4cad-413e-9225-95cb38cfc404\") " Oct 30 21:50:30.174214 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.174186 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca" pod "ff33b735-4cad-413e-9225-95cb38cfc404" (UID: "ff33b735-4cad-413e-9225-95cb38cfc404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:30.175788 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.175764 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca" pod "ff33b735-4cad-413e-9225-95cb38cfc404" (UID: "ff33b735-4cad-413e-9225-95cb38cfc404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:30.175899 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.175841 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml" pod "ff33b735-4cad-413e-9225-95cb38cfc404" (UID: "ff33b735-4cad-413e-9225-95cb38cfc404"). InnerVolumeSpecName "kube-api-access-62dml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:30.188282 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.188259 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-be9451/distinct-w0-be9451-kserve-6ccccdc5cf-fnchh"] Oct 30 21:50:30.275472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.275401 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.275472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.275425 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dml\" (UniqueName: \"kubernetes.io/projected/ff33b735-4cad-413e-9225-95cb38cfc404-kube-api-access-62dml\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.275472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.275434 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff33b735-4cad-413e-9225-95cb38cfc404-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:50:30.986043 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.986017 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" Oct 30 21:50:30.986451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.986061 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq" event={"ID":"ff33b735-4cad-413e-9225-95cb38cfc404","Type":"ContainerDied","Data":"9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be"} Oct 30 21:50:30.986451 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:30.986113 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9872932009ba7c9f9ae3175dd33dce2c524c2b1cf470cd5127ecba8749ba14be" Oct 30 21:50:32.167496 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:32.167457 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" path="/var/lib/kubelet/pods/3eb422e0-1506-4ccb-84d1-e29858c3b9cd/volumes" Oct 30 21:50:36.552283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552249 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:50:36.552886 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552869 4921 cpu_manager.go:414] "containerMap: removing container" podUID="ff33b735-4cad-413e-9225-95cb38cfc404" containerName="revoke-keys" Oct 30 21:50:36.552935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552890 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff33b735-4cad-413e-9225-95cb38cfc404" containerName="revoke-keys" Oct 30 21:50:36.552935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552904 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" Oct 30 21:50:36.552935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552908 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" Oct 30 21:50:36.553031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.552992 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff33b735-4cad-413e-9225-95cb38cfc404" containerName="revoke-keys" Oct 30 21:50:36.553031 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.553001 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb422e0-1506-4ccb-84d1-e29858c3b9cd" containerName="main" Oct 30 21:50:36.556426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.556411 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.607935 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.607911 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3ab88248-serving-cert\"" Oct 30 21:50:36.620544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.620519 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:50:36.637406 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.637382 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.637502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.637427 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.637684 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.637662 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.637727 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.637708 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sl7f\" (UniqueName: \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.738389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.738354 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.738528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.738398 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sl7f\" (UniqueName: \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.738528 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.738460 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.738889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.738504 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.739053 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:36.739034 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-serving-cert: secret "maas-api-e2e-mt-3ab88248-serving-cert" not found Oct 30 21:50:36.739122 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:36.739104 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls podName:b230b839-f794-4362-84e5-82cd77fd2b4e nodeName:}" failed. No retries permitted until 2026-10-30 21:50:37.239085598 +0000 UTC m=+2351.221786013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls") pod "maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" (UID: "b230b839-f794-4362-84e5-82cd77fd2b4e") : secret "maas-api-e2e-mt-3ab88248-serving-cert" not found Oct 30 21:50:36.739227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.739206 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.741269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.741243 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:36.806378 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:36.806311 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sl7f\" (UniqueName: \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:37.246368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:37.246339 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:37.248594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:37.248574 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") pod \"maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:37.469254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:37.469231 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:37.560389 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:50:37.560330 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:50:37.848249 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:37.848180 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:50:37.851578 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:50:37.851544 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb230b839_f794_4362_84e5_82cd77fd2b4e.slice/crio-1018315c746a9503250bf112e99435cd16172dee521d70a06077f7fbf013b3be WatchSource:0}: Error finding container 1018315c746a9503250bf112e99435cd16172dee521d70a06077f7fbf013b3be: Status 404 returned error can't find the container with id 1018315c746a9503250bf112e99435cd16172dee521d70a06077f7fbf013b3be Oct 30 21:50:38.035664 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:38.035633 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" event={"ID":"b230b839-f794-4362-84e5-82cd77fd2b4e","Type":"ContainerStarted","Data":"1018315c746a9503250bf112e99435cd16172dee521d70a06077f7fbf013b3be"} Oct 30 21:50:39.045291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:39.045249 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" event={"ID":"b230b839-f794-4362-84e5-82cd77fd2b4e","Type":"ContainerStarted","Data":"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3"} Oct 30 21:50:39.045704 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:39.045377 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:50:39.162505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:39.162462 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" podStartSLOduration=2.328274788 podStartE2EDuration="3.162447568s" totalImagesPullingTime="834.17278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:39.158868727 +0000 UTC m=+2353.141569145" watchObservedRunningTime="2026-10-30 21:50:39.162447568 +0000 UTC m=+2353.145147979" Oct 30 21:50:39.979751 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:39.979722 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:50:39.979940 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:39.979785 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:50:45.057862 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:50:45.057830 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:51:01.714929 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:01.714894 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:51:01.715446 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:01.715076 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" containerID="cri-o://637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb" gracePeriod=30 Oct 30 21:51:06.977107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:06.977083 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:51:07.167185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167115 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167165 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167200 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167298 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167403 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167377 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167409 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167430 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167489 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167513 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\") pod \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\" (UID: \"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac\") " Oct 30 21:51:07.167984 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.167963 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:07.168509 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.168487 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:07.168715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.168548 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:07.168715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.168602 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:07.168715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.168701 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:07.170060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.170031 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:07.170159 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.170132 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:07.170369 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.170347 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:07.170998 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.170980 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv" pod "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" (UID: "0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac"). InnerVolumeSpecName "kube-api-access-l84mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:07.228359 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.228327 4921 generic.go:356] "Generic (PLEG): container finished" podID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerID="637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb" exitCode=0 Oct 30 21:51:07.228458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.228377 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" Oct 30 21:51:07.228458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.228400 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerDied","Data":"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb"} Oct 30 21:51:07.228458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.228455 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj" event={"ID":"0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac","Type":"ContainerDied","Data":"79ead91c378d86d6d251e06227aede9392c17ffddbcf119460d7478dbc29fee6"} Oct 30 21:51:07.228579 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.228476 4921 scope.go:122] "RemoveContainer" containerID="637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb" Oct 30 21:51:07.238845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.238826 4921 scope.go:122] "RemoveContainer" containerID="dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf" Oct 30 21:51:07.247815 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.247796 4921 scope.go:122] "RemoveContainer" containerID="637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb" Oct 30 21:51:07.248049 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:51:07.248031 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb\": container with ID starting with 637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb not found: ID does not exist" containerID="637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb" Oct 30 21:51:07.248111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.248056 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb"} err="failed to get container status \"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb\": rpc error: code = NotFound desc = could not find container \"637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb\": container with ID starting with 637e0c5517796899a8b54bedde77ef144ec15f722f2e329fe14208cc40b316eb not found: ID does not exist" Oct 30 21:51:07.248111 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.248071 4921 scope.go:122] "RemoveContainer" containerID="dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf" Oct 30 21:51:07.248352 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:51:07.248303 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf\": container with ID starting with dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf not found: ID does not exist" containerID="dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf" Oct 30 21:51:07.248420 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.248363 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf"} err="failed to get container status \"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf\": rpc error: code = NotFound desc = could not find container \"dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf\": container with ID starting with dfe8f04a4c94667917511a4630eea9c17e3d599b5618f6e72a78fbbe8dbb77cf not found: ID does not exist" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269390 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84mv\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-kube-api-access-l84mv\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269419 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269434 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269448 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269462 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269475 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269488 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269503 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.269633 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.269517 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:07.390655 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.390620 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:51:07.393921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:07.393895 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8ce31a-openshift-default-bc8b5cfb5-8rvxj"] Oct 30 21:51:07.540189 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:51:07.540158 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:51:08.168160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:08.168130 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" path="/var/lib/kubelet/pods/0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac/volumes" Oct 30 21:51:17.013889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:17.013856 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:51:17.014287 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:17.014050 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" containerID="cri-o://4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b" gracePeriod=30 Oct 30 21:51:17.062685 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:17.062651 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:51:17.062884 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:17.062859 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" containerID="cri-o://f422bc0711937369f7e8cc854a4b12f37209ebe144d57775916d775a0c5d6ce6" gracePeriod=30 Oct 30 21:51:20.865856 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:20.865813 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2vlfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:20.866209 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:20.865882 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:21.039076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:21.039043 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-8l67h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:21.039233 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:21.039091 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:22.308219 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.308195 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:51:22.319487 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.319458 4921 generic.go:356] "Generic (PLEG): container finished" podID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerID="f422bc0711937369f7e8cc854a4b12f37209ebe144d57775916d775a0c5d6ce6" exitCode=0 Oct 30 21:51:22.319594 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.319524 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" event={"ID":"9f4ca85d-68a4-46e4-9198-2c803b86742d","Type":"ContainerDied","Data":"f422bc0711937369f7e8cc854a4b12f37209ebe144d57775916d775a0c5d6ce6"} Oct 30 21:51:22.320889 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.320870 4921 generic.go:356] "Generic (PLEG): container finished" podID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerID="4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b" exitCode=0 Oct 30 21:51:22.320985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.320939 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" Oct 30 21:51:22.320985 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.320949 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" event={"ID":"eb774724-92b3-48f7-8d62-fc5c321ad16c","Type":"ContainerDied","Data":"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b"} Oct 30 21:51:22.321052 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.321000 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj" event={"ID":"eb774724-92b3-48f7-8d62-fc5c321ad16c","Type":"ContainerDied","Data":"6c561d7902b7ed95ac5507fb7973dc1e0c2591cfb654a2eb4ea726619c51f709"} Oct 30 21:51:22.321052 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.321021 4921 scope.go:122] "RemoveContainer" containerID="4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b" Oct 30 21:51:22.331606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.331533 4921 scope.go:122] "RemoveContainer" containerID="4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b" Oct 30 21:51:22.332183 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:51:22.332113 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b\": container with ID starting with 4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b not found: ID does not exist" containerID="4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b" Oct 30 21:51:22.332183 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.332146 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b"} err="failed to get container status \"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b\": rpc error: code = NotFound desc = could not find container \"4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b\": container with ID starting with 4cbd61eb11702ce396988440ac1d23b82ab7ca27049c3be537c9481b23e3b29b not found: ID does not exist" Oct 30 21:51:22.351100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.351085 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:51:22.412954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.412924 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.412981 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413028 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413079 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413070 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413114 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413145 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413185 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413220 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413259 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413250 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413279 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413336 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413365 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413403 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413417 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413430 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413548 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413502 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\") pod \"9f4ca85d-68a4-46e4-9198-2c803b86742d\" (UID: \"9f4ca85d-68a4-46e4-9198-2c803b86742d\") " Oct 30 21:51:22.413921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413650 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413685 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.413921 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.413720 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\") pod \"eb774724-92b3-48f7-8d62-fc5c321ad16c\" (UID: \"eb774724-92b3-48f7-8d62-fc5c321ad16c\") " Oct 30 21:51:22.414151 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.414132 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb774724-92b3-48f7-8d62-fc5c321ad16c-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.414382 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.414593 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.414909 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415048 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415172 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415258 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415454 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415616 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415637 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.415907 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.416400 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.416228 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "kube-api-access-hr54l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.417733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.417708 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:22.417733 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.417723 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:22.418117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.418095 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "kube-api-access-65k4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.418689 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.418665 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token" pod "9f4ca85d-68a4-46e4-9198-2c803b86742d" (UID: "9f4ca85d-68a4-46e4-9198-2c803b86742d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.419072 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.419051 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.419981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.419943 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token" pod "eb774724-92b3-48f7-8d62-fc5c321ad16c" (UID: "eb774724-92b3-48f7-8d62-fc5c321ad16c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.515055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515024 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65k4g\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-kube-api-access-65k4g\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515055 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515054 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515068 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4ca85d-68a4-46e4-9198-2c803b86742d-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515081 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515093 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515106 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515118 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515131 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515144 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515157 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr54l\" (UniqueName: \"kubernetes.io/projected/9f4ca85d-68a4-46e4-9198-2c803b86742d-kube-api-access-hr54l\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515170 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515183 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515198 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515212 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4ca85d-68a4-46e4-9198-2c803b86742d-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515222 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515225 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515239 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.515659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.515253 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb774724-92b3-48f7-8d62-fc5c321ad16c-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.755535 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.755503 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:51:22.759291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:22.759267 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2vlfj"] Oct 30 21:51:23.328917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:23.328852 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" event={"ID":"9f4ca85d-68a4-46e4-9198-2c803b86742d","Type":"ContainerDied","Data":"ef49d40de29993bc4fdef8942f95c800c9f14e0d522f888dec577e3e18fbf386"} Oct 30 21:51:23.328917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:23.328872 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h" Oct 30 21:51:23.328917 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:23.328909 4921 scope.go:122] "RemoveContainer" containerID="f422bc0711937369f7e8cc854a4b12f37209ebe144d57775916d775a0c5d6ce6" Oct 30 21:51:23.491242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:23.491219 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:51:23.540703 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:23.540682 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-8l67h"] Oct 30 21:51:24.169695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:24.169654 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" path="/var/lib/kubelet/pods/9f4ca85d-68a4-46e4-9198-2c803b86742d/volumes" Oct 30 21:51:24.170257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:24.170236 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" path="/var/lib/kubelet/pods/eb774724-92b3-48f7-8d62-fc5c321ad16c/volumes" Oct 30 21:51:26.803981 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:26.803956 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:51:26.814674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:26.814655 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:51:26.859273 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:26.859252 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:51:26.870138 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:51:26.870116 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:51:37.538425 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:51:37.538388 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:52:07.538604 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:07.538571 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:52:29.736859 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:29.736828 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:52:29.841969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:29.841933 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:52:34.737501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:34.737467 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" podUID="3ca84a6d-8e36-4939-8001-aabd34323020" containerName="payload-processing" containerID="cri-o://7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948" gracePeriod=25 Oct 30 21:52:34.842307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:34.842270 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" podUID="26a060e7-4256-482e-ad22-607dfe1acbca" containerName="payload-pre-processing" containerID="cri-o://793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d" gracePeriod=25 Oct 30 21:52:34.984978 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:34.984954 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:52:35.099291 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.099262 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:52:35.149046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.149020 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\" (UniqueName: \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\") pod \"3ca84a6d-8e36-4939-8001-aabd34323020\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " Oct 30 21:52:35.149169 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.149069 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\") pod \"3ca84a6d-8e36-4939-8001-aabd34323020\" (UID: \"3ca84a6d-8e36-4939-8001-aabd34323020\") " Oct 30 21:52:35.149478 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.149453 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume" pod "3ca84a6d-8e36-4939-8001-aabd34323020" (UID: "3ca84a6d-8e36-4939-8001-aabd34323020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:35.151213 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.151194 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz" pod "3ca84a6d-8e36-4939-8001-aabd34323020" (UID: "3ca84a6d-8e36-4939-8001-aabd34323020"). InnerVolumeSpecName "kube-api-access-qdpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:35.250243 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.250220 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\" (UniqueName: \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\") pod \"26a060e7-4256-482e-ad22-607dfe1acbca\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " Oct 30 21:52:35.250379 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.250256 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\") pod \"26a060e7-4256-482e-ad22-607dfe1acbca\" (UID: \"26a060e7-4256-482e-ad22-607dfe1acbca\") " Oct 30 21:52:35.250558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.250546 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdpqz\" (UniqueName: \"kubernetes.io/projected/3ca84a6d-8e36-4939-8001-aabd34323020-kube-api-access-qdpqz\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:35.250593 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.250560 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca84a6d-8e36-4939-8001-aabd34323020-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:35.250639 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.250621 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume" pod "26a060e7-4256-482e-ad22-607dfe1acbca" (UID: "26a060e7-4256-482e-ad22-607dfe1acbca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:35.252100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.252081 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w" pod "26a060e7-4256-482e-ad22-607dfe1acbca" (UID: "26a060e7-4256-482e-ad22-607dfe1acbca"). InnerVolumeSpecName "kube-api-access-v2m2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:35.351913 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.351866 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2m2w\" (UniqueName: \"kubernetes.io/projected/26a060e7-4256-482e-ad22-607dfe1acbca-kube-api-access-v2m2w\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:35.351913 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.351885 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a060e7-4256-482e-ad22-607dfe1acbca-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:35.754085 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.754052 4921 generic.go:356] "Generic (PLEG): container finished" podID="26a060e7-4256-482e-ad22-607dfe1acbca" containerID="793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d" exitCode=0 Oct 30 21:52:35.754486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.754102 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" Oct 30 21:52:35.754486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.754139 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" event={"ID":"26a060e7-4256-482e-ad22-607dfe1acbca","Type":"ContainerDied","Data":"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d"} Oct 30 21:52:35.754486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.754189 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn" event={"ID":"26a060e7-4256-482e-ad22-607dfe1acbca","Type":"ContainerDied","Data":"60da011c6583cf30d2e85c178a7b387bf7d5e849788c7feb753fcb5e6f4224c6"} Oct 30 21:52:35.754486 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.754211 4921 scope.go:122] "RemoveContainer" containerID="793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d" Oct 30 21:52:35.755514 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.755492 4921 generic.go:356] "Generic (PLEG): container finished" podID="3ca84a6d-8e36-4939-8001-aabd34323020" containerID="7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948" exitCode=0 Oct 30 21:52:35.755605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.755537 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" Oct 30 21:52:35.755605 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.755582 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" event={"ID":"3ca84a6d-8e36-4939-8001-aabd34323020","Type":"ContainerDied","Data":"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948"} Oct 30 21:52:35.755717 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.755627 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p" event={"ID":"3ca84a6d-8e36-4939-8001-aabd34323020","Type":"ContainerDied","Data":"cfdf062e9bc1a2146dd26e4dddc848e783b4b1c7e16a7933e31d68e1db2ca93a"} Oct 30 21:52:35.763631 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.763613 4921 scope.go:122] "RemoveContainer" containerID="793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d" Oct 30 21:52:35.763884 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:35.763866 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d\": container with ID starting with 793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d not found: ID does not exist" containerID="793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d" Oct 30 21:52:35.763967 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.763895 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d"} err="failed to get container status \"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d\": rpc error: code = NotFound desc = could not find container \"793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d\": container with ID starting with 793164d5e9a5f0b1a5b607c80aec8eb827ee0dee045ef312cb20b95ec1a7029d not found: ID does not exist" Oct 30 21:52:35.763967 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.763917 4921 scope.go:122] "RemoveContainer" containerID="7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948" Oct 30 21:52:35.772116 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.772098 4921 scope.go:122] "RemoveContainer" containerID="7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948" Oct 30 21:52:35.772418 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:35.772398 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948\": container with ID starting with 7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948 not found: ID does not exist" containerID="7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948" Oct 30 21:52:35.772477 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.772424 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948"} err="failed to get container status \"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948\": rpc error: code = NotFound desc = could not find container \"7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948\": container with ID starting with 7bec13628369ef8d52365a2b4258603d5641f01f36c4b2dfcdc6f514a9052948 not found: ID does not exist" Oct 30 21:52:35.917536 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.917511 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:52:35.966788 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:35.966768 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ab88248-58b75f5f87-lf5tn"] Oct 30 21:52:36.028694 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.028645 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:52:36.032808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.032789 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ab88248-7f6d44cc95-vt77p"] Oct 30 21:52:36.068969 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.068949 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:52:36.069155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.069135 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" podUID="b230b839-f794-4362-84e5-82cd77fd2b4e" containerName="maas-api" containerID="cri-o://d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3" gracePeriod=30 Oct 30 21:52:36.167410 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.167383 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a060e7-4256-482e-ad22-607dfe1acbca" path="/var/lib/kubelet/pods/26a060e7-4256-482e-ad22-607dfe1acbca/volumes" Oct 30 21:52:36.167745 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.167732 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca84a6d-8e36-4939-8001-aabd34323020" path="/var/lib/kubelet/pods/3ca84a6d-8e36-4939-8001-aabd34323020/volumes" Oct 30 21:52:36.328437 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.328413 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:52:36.461551 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.461520 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") pod \"b230b839-f794-4362-84e5-82cd77fd2b4e\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " Oct 30 21:52:36.461716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.461578 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\") pod \"b230b839-f794-4362-84e5-82cd77fd2b4e\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " Oct 30 21:52:36.461716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.461644 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\") pod \"b230b839-f794-4362-84e5-82cd77fd2b4e\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " Oct 30 21:52:36.461716 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.461705 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\" (UniqueName: \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\") pod \"b230b839-f794-4362-84e5-82cd77fd2b4e\" (UID: \"b230b839-f794-4362-84e5-82cd77fd2b4e\") " Oct 30 21:52:36.462096 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.462062 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca" pod "b230b839-f794-4362-84e5-82cd77fd2b4e" (UID: "b230b839-f794-4362-84e5-82cd77fd2b4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:36.463695 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.463665 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls" pod "b230b839-f794-4362-84e5-82cd77fd2b4e" (UID: "b230b839-f794-4362-84e5-82cd77fd2b4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:36.463809 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.463693 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs" pod "b230b839-f794-4362-84e5-82cd77fd2b4e" (UID: "b230b839-f794-4362-84e5-82cd77fd2b4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:36.464109 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.464090 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f" pod "b230b839-f794-4362-84e5-82cd77fd2b4e" (UID: "b230b839-f794-4362-84e5-82cd77fd2b4e"). InnerVolumeSpecName "kube-api-access-5sl7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:36.563744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.563643 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sl7f\" (UniqueName: \"kubernetes.io/projected/b230b839-f794-4362-84e5-82cd77fd2b4e-kube-api-access-5sl7f\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:36.563744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.563708 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:36.563744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.563724 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b230b839-f794-4362-84e5-82cd77fd2b4e-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:36.563744 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.563737 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b230b839-f794-4362-84e5-82cd77fd2b4e-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:36.765673 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.765645 4921 generic.go:356] "Generic (PLEG): container finished" podID="b230b839-f794-4362-84e5-82cd77fd2b4e" containerID="d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3" exitCode=0 Oct 30 21:52:36.766083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.765690 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" Oct 30 21:52:36.766083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.765702 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" event={"ID":"b230b839-f794-4362-84e5-82cd77fd2b4e","Type":"ContainerDied","Data":"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3"} Oct 30 21:52:36.766083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.765741 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr" event={"ID":"b230b839-f794-4362-84e5-82cd77fd2b4e","Type":"ContainerDied","Data":"1018315c746a9503250bf112e99435cd16172dee521d70a06077f7fbf013b3be"} Oct 30 21:52:36.766083 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.765760 4921 scope.go:122] "RemoveContainer" containerID="d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3" Oct 30 21:52:36.776033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.776013 4921 scope.go:122] "RemoveContainer" containerID="d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3" Oct 30 21:52:36.776298 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:36.776280 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3\": container with ID starting with d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3 not found: ID does not exist" containerID="d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3" Oct 30 21:52:36.776365 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.776305 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3"} err="failed to get container status \"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3\": rpc error: code = NotFound desc = could not find container \"d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3\": container with ID starting with d50a76e94fed3f95bf922591b22e8c058d8363425ad0bcfa159a831ddfc934b3 not found: ID does not exist" Oct 30 21:52:36.934840 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.934744 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:52:36.936860 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:36.936837 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3ab88248-bd5c8d54b-96fxr"] Oct 30 21:52:37.538932 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:37.538899 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:52:38.168472 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:38.168441 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b230b839-f794-4362-84e5-82cd77fd2b4e" path="/var/lib/kubelet/pods/b230b839-f794-4362-84e5-82cd77fd2b4e/volumes" Oct 30 21:52:43.599784 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:43.599751 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:52:58.619331 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.619282 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" podUID="55f436a8-12af-44e0-9d74-95575a49155d" containerName="main" containerID="cri-o://a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d" gracePeriod=45 Oct 30 21:52:58.871678 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.871624 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:52:58.902346 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.902294 4921 generic.go:356] "Generic (PLEG): container finished" podID="55f436a8-12af-44e0-9d74-95575a49155d" containerID="a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d" exitCode=0 Oct 30 21:52:58.902449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.902351 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" Oct 30 21:52:58.902449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.902384 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" event={"ID":"55f436a8-12af-44e0-9d74-95575a49155d","Type":"ContainerDied","Data":"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d"} Oct 30 21:52:58.902449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.902424 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm" event={"ID":"55f436a8-12af-44e0-9d74-95575a49155d","Type":"ContainerDied","Data":"3efb7b0a4e42d861f7172f4f30ddefabfab40670887cc8a698ad9686d2bd7394"} Oct 30 21:52:58.902449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.902442 4921 scope.go:122] "RemoveContainer" containerID="a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d" Oct 30 21:52:58.911212 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.911197 4921 scope.go:122] "RemoveContainer" containerID="a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d" Oct 30 21:52:58.911539 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:52:58.911519 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d\": container with ID starting with a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d not found: ID does not exist" containerID="a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d" Oct 30 21:52:58.911606 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.911550 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d"} err="failed to get container status \"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d\": rpc error: code = NotFound desc = could not find container \"a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d\": container with ID starting with a4917a05127ae1369d95c932c30f99dbbb09cd19c5fb0e5647d84f84b4aabd1d not found: ID does not exist" Oct 30 21:52:58.976949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.976920 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.976982 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\" (UniqueName: \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977048 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977094 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977123 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977192 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977186 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\" (UniqueName: \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\") pod \"55f436a8-12af-44e0-9d74-95575a49155d\" (UID: \"55f436a8-12af-44e0-9d74-95575a49155d\") " Oct 30 21:52:58.977381 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977188 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:58.977454 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977390 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:58.977506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977456 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:58.977638 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977615 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:58.977753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977641 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:58.977753 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.977654 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:58.979345 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.979307 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:58.979448 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.979432 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:58.979494 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:58.979462 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf" pod "55f436a8-12af-44e0-9d74-95575a49155d" (UID: "55f436a8-12af-44e0-9d74-95575a49155d"). InnerVolumeSpecName "kube-api-access-qx6qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:59.078665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:59.078639 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f436a8-12af-44e0-9d74-95575a49155d-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:59.078665 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:59.078664 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6qf\" (UniqueName: \"kubernetes.io/projected/55f436a8-12af-44e0-9d74-95575a49155d-kube-api-access-qx6qf\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:59.078842 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:59.078674 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f436a8-12af-44e0-9d74-95575a49155d-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:59.328434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:59.328363 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:52:59.329510 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:52:59.329480 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3ab88248/e2e-lifecycle-model-3ab88248-kserve-df8db6cf7-qndrm"] Oct 30 21:53:00.167121 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:00.167092 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f436a8-12af-44e0-9d74-95575a49155d" path="/var/lib/kubelet/pods/55f436a8-12af-44e0-9d74-95575a49155d/volumes" Oct 30 21:53:07.539298 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:53:07.539261 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:53:27.900226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:27.900133 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:53:27.900643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:27.900355 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" containerID="cri-o://d772a6e33ca5df4c979181f37a676e45a3c879b06063f1abfefbbf2e195a6e47" gracePeriod=30 Oct 30 21:53:28.398455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:28.398422 4921 patch_prober.go:29] interesting pod/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:53:28.398620 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:28.398483 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:53:33.110529 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.110493 4921 generic.go:356] "Generic (PLEG): container finished" podID="8b07a77a-abde-4656-a9cb-539187d89e50" containerID="d772a6e33ca5df4c979181f37a676e45a3c879b06063f1abfefbbf2e195a6e47" exitCode=0 Oct 30 21:53:33.110977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.110568 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerDied","Data":"d772a6e33ca5df4c979181f37a676e45a3c879b06063f1abfefbbf2e195a6e47"} Oct 30 21:53:33.110977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.110644 4921 scope.go:122] "RemoveContainer" containerID="f411ff91e86e6845834c50379174f3b64b7504fc2511145d7dfa7913af0354da" Oct 30 21:53:33.163254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.163202 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:53:33.264587 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264563 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264602 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264630 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264723 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264654 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264725 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264756 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264829 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264861 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.264903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264889 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\") pod \"8b07a77a-abde-4656-a9cb-539187d89e50\" (UID: \"8b07a77a-abde-4656-a9cb-539187d89e50\") " Oct 30 21:53:33.265172 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.264910 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:33.265231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.265195 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:33.265231 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.265211 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:33.265371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.265227 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.265371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.265271 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:33.265519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.265501 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:33.266706 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.266685 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "kube-api-access-phh5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:33.266828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.266793 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:33.267101 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.267080 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:53:33.267101 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.267090 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy" pod "8b07a77a-abde-4656-a9cb-539187d89e50" (UID: "8b07a77a-abde-4656-a9cb-539187d89e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:33.366265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366238 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366268 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366281 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b07a77a-abde-4656-a9cb-539187d89e50-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366295 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366309 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b07a77a-abde-4656-a9cb-539187d89e50-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366337 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b07a77a-abde-4656-a9cb-539187d89e50-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366364 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366352 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh5p\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-kube-api-access-phh5p\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:33.366668 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:33.366367 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b07a77a-abde-4656-a9cb-539187d89e50-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:34.118242 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:34.118220 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" Oct 30 21:53:34.118656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:34.118215 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf" event={"ID":"8b07a77a-abde-4656-a9cb-539187d89e50","Type":"ContainerDied","Data":"3e5c72ea10c7db14437516c2ab842f57dca944dc6eb5cd32b8a6972bbdf50d25"} Oct 30 21:53:34.118656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:34.118394 4921 scope.go:122] "RemoveContainer" containerID="d772a6e33ca5df4c979181f37a676e45a3c879b06063f1abfefbbf2e195a6e47" Oct 30 21:53:34.281158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:34.281132 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:53:34.330906 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:34.330881 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3ab88248-openshift-default-8c7c5d5c7-jznlf"] Oct 30 21:53:36.166473 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:36.166439 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" path="/var/lib/kubelet/pods/8b07a77a-abde-4656-a9cb-539187d89e50/volumes" Oct 30 21:53:37.538168 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:53:37.538132 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:53:53.563274 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.563240 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:53:53.563763 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.563429 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" podUID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" containerName="maas-api" containerID="cri-o://b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208" gracePeriod=30 Oct 30 21:53:53.799850 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.799826 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:53:53.827245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827191 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\") pod \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " Oct 30 21:53:53.827396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827267 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\") pod \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " Oct 30 21:53:53.827396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827302 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\" (UniqueName: \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\") pod \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " Oct 30 21:53:53.827396 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827363 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") pod \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\" (UID: \"c345b684-a3f9-4bdf-9d7d-7a22b56067df\") " Oct 30 21:53:53.827601 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827575 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca" pod "c345b684-a3f9-4bdf-9d7d-7a22b56067df" (UID: "c345b684-a3f9-4bdf-9d7d-7a22b56067df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:53.827813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.827788 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c345b684-a3f9-4bdf-9d7d-7a22b56067df-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:53.829353 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.829327 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls" pod "c345b684-a3f9-4bdf-9d7d-7a22b56067df" (UID: "c345b684-a3f9-4bdf-9d7d-7a22b56067df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:53.829740 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.829719 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs" pod "c345b684-a3f9-4bdf-9d7d-7a22b56067df" (UID: "c345b684-a3f9-4bdf-9d7d-7a22b56067df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:53.829805 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.829746 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv" pod "c345b684-a3f9-4bdf-9d7d-7a22b56067df" (UID: "c345b684-a3f9-4bdf-9d7d-7a22b56067df"). InnerVolumeSpecName "kube-api-access-z8fxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:53.929069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.929049 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:53.929069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.929069 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c345b684-a3f9-4bdf-9d7d-7a22b56067df-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:53.929227 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:53.929080 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fxv\" (UniqueName: \"kubernetes.io/projected/c345b684-a3f9-4bdf-9d7d-7a22b56067df-kube-api-access-z8fxv\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:54.232875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.232845 4921 generic.go:356] "Generic (PLEG): container finished" podID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" containerID="b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208" exitCode=0 Oct 30 21:53:54.233033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.232884 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" Oct 30 21:53:54.233033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.232937 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" event={"ID":"c345b684-a3f9-4bdf-9d7d-7a22b56067df","Type":"ContainerDied","Data":"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208"} Oct 30 21:53:54.233033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.232993 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl" event={"ID":"c345b684-a3f9-4bdf-9d7d-7a22b56067df","Type":"ContainerDied","Data":"c0377ef60c459276d91d034949c30d2091f6ed6b95b64b14a4e31c801d670695"} Oct 30 21:53:54.233033 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.233021 4921 scope.go:122] "RemoveContainer" containerID="b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208" Oct 30 21:53:54.242107 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.242091 4921 scope.go:122] "RemoveContainer" containerID="b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208" Oct 30 21:53:54.242369 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:53:54.242352 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208\": container with ID starting with b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208 not found: ID does not exist" containerID="b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208" Oct 30 21:53:54.242427 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.242378 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208"} err="failed to get container status \"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208\": rpc error: code = NotFound desc = could not find container \"b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208\": container with ID starting with b716c8c77abf8da352b223dc5598ee815842484eee4ebdab62c098496634c208 not found: ID does not exist" Oct 30 21:53:54.396271 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.396249 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:53:54.401463 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:54.401443 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-be8b73-5f974d6c6d-jdjvl"] Oct 30 21:53:56.167401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:53:56.167360 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" path="/var/lib/kubelet/pods/c345b684-a3f9-4bdf-9d7d-7a22b56067df/volumes" Oct 30 21:54:03.731090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731054 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv"] Oct 30 21:54:03.731531 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731517 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:03.731572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731532 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:03.731572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731545 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b230b839-f794-4362-84e5-82cd77fd2b4e" containerName="maas-api" Oct 30 21:54:03.731572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731550 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b230b839-f794-4362-84e5-82cd77fd2b4e" containerName="maas-api" Oct 30 21:54:03.731572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731560 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" containerName="maas-api" Oct 30 21:54:03.731572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731567 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" containerName="maas-api" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731579 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731584 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731597 4921 cpu_manager.go:414] "containerMap: removing container" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731602 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731613 4921 cpu_manager.go:414] "containerMap: removing container" podUID="26a060e7-4256-482e-ad22-607dfe1acbca" containerName="payload-pre-processing" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731618 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a060e7-4256-482e-ad22-607dfe1acbca" containerName="payload-pre-processing" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731630 4921 cpu_manager.go:414] "containerMap: removing container" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731634 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731642 4921 cpu_manager.go:414] "containerMap: removing container" podUID="3ca84a6d-8e36-4939-8001-aabd34323020" containerName="payload-processing" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731649 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca84a6d-8e36-4939-8001-aabd34323020" containerName="payload-processing" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731661 4921 cpu_manager.go:414] "containerMap: removing container" podUID="55f436a8-12af-44e0-9d74-95575a49155d" containerName="main" Oct 30 21:54:03.731725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731665 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f436a8-12af-44e0-9d74-95575a49155d" containerName="main" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731745 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a060e7-4256-482e-ad22-607dfe1acbca" containerName="payload-pre-processing" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731756 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb774724-92b3-48f7-8d62-fc5c321ad16c" containerName="istio-proxy" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731762 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca84a6d-8e36-4939-8001-aabd34323020" containerName="payload-processing" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731768 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731774 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f436a8-12af-44e0-9d74-95575a49155d" containerName="main" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731781 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731786 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b230b839-f794-4362-84e5-82cd77fd2b4e" containerName="maas-api" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731792 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c345b684-a3f9-4bdf-9d7d-7a22b56067df" containerName="maas-api" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731797 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:03.732076 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.731803 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4ca85d-68a4-46e4-9198-2c803b86742d" containerName="istio-proxy" Oct 30 21:54:03.736247 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.736229 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.819625 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.819591 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.819754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.819669 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmpj\" (UniqueName: \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.819754 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.819730 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.837675 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.837654 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:03.837843 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.837679 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:54:03.840395 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.840374 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv"] Oct 30 21:54:03.920185 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.920157 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmpj\" (UniqueName: \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.920298 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.920203 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.920298 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.920264 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.920832 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.920814 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.922524 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.922504 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:03.976143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:03.976123 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmpj\" (UniqueName: \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:04.049595 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:04.049546 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:04.422545 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:04.422512 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv"] Oct 30 21:54:04.423729 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:54:04.423705 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc865302a_0e53_4c41_aac6_5064c4243e14.slice/crio-b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892 WatchSource:0}: Error finding container b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892: Status 404 returned error can't find the container with id b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892 Oct 30 21:54:05.300873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:05.300834 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" event={"ID":"c865302a-0e53-4c41-aac6-5064c4243e14","Type":"ContainerStarted","Data":"e9687abd15f7168f747100ff057734ad6bdfb1f7a803dc64638b635ba34e1cf7"} Oct 30 21:54:05.301295 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:05.300921 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" event={"ID":"c865302a-0e53-4c41-aac6-5064c4243e14","Type":"ContainerStarted","Data":"b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892"} Oct 30 21:54:05.414790 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:05.414744 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" podStartSLOduration=2.414731109 podStartE2EDuration="2.414731109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.413152463 +0000 UTC m=+2559.395852881" watchObservedRunningTime="2026-10-30 21:54:05.414731109 +0000 UTC m=+2559.397431527" Oct 30 21:54:06.308680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:06.308644 4921 generic.go:356] "Generic (PLEG): container finished" podID="c865302a-0e53-4c41-aac6-5064c4243e14" containerID="e9687abd15f7168f747100ff057734ad6bdfb1f7a803dc64638b635ba34e1cf7" exitCode=0 Oct 30 21:54:06.309089 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:06.308740 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" event={"ID":"c865302a-0e53-4c41-aac6-5064c4243e14","Type":"ContainerDied","Data":"e9687abd15f7168f747100ff057734ad6bdfb1f7a803dc64638b635ba34e1cf7"} Oct 30 21:54:07.442559 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.442530 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:07.538220 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:54:07.538191 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:54:07.550900 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.550877 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\" (UniqueName: \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\") pod \"c865302a-0e53-4c41-aac6-5064c4243e14\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " Oct 30 21:54:07.551021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.550939 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\") pod \"c865302a-0e53-4c41-aac6-5064c4243e14\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " Oct 30 21:54:07.551021 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.551012 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\") pod \"c865302a-0e53-4c41-aac6-5064c4243e14\" (UID: \"c865302a-0e53-4c41-aac6-5064c4243e14\") " Oct 30 21:54:07.551398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.551378 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca" pod "c865302a-0e53-4c41-aac6-5064c4243e14" (UID: "c865302a-0e53-4c41-aac6-5064c4243e14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:07.552797 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.552773 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj" pod "c865302a-0e53-4c41-aac6-5064c4243e14" (UID: "c865302a-0e53-4c41-aac6-5064c4243e14"). InnerVolumeSpecName "kube-api-access-zzmpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:07.552898 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.552879 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca" pod "c865302a-0e53-4c41-aac6-5064c4243e14" (UID: "c865302a-0e53-4c41-aac6-5064c4243e14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:07.651845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.651783 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.651845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.651807 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c865302a-0e53-4c41-aac6-5064c4243e14-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.651845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:07.651817 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmpj\" (UniqueName: \"kubernetes.io/projected/c865302a-0e53-4c41-aac6-5064c4243e14-kube-api-access-zzmpj\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:08.321283 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:08.321254 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" Oct 30 21:54:08.321458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:08.321298 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv" event={"ID":"c865302a-0e53-4c41-aac6-5064c4243e14","Type":"ContainerDied","Data":"b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892"} Oct 30 21:54:08.321458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:08.321369 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b339bcba06df10e7599be5f16339492b2a0acb018bea8f935a65e2cfa13e4892" Oct 30 21:54:25.925951 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:25.925911 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv"] Oct 30 21:54:25.974916 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:25.974887 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-5f31bd-695rv"] Oct 30 21:54:26.167558 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:26.167527 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c865302a-0e53-4c41-aac6-5064c4243e14" path="/var/lib/kubelet/pods/c865302a-0e53-4c41-aac6-5064c4243e14/volumes" Oct 30 21:54:37.539295 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:54:37.539255 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:54:38.768384 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:38.768343 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:54:38.817874 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:38.817842 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:54:45.315470 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.315440 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx"] Oct 30 21:54:45.316020 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.315999 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:45.316108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316022 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:45.316108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316047 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:45.316108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316055 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b07a77a-abde-4656-a9cb-539187d89e50" containerName="istio-proxy" Oct 30 21:54:45.316108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316085 4921 cpu_manager.go:414] "containerMap: removing container" podUID="c865302a-0e53-4c41-aac6-5064c4243e14" containerName="revoke-keys" Oct 30 21:54:45.316108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316092 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c865302a-0e53-4c41-aac6-5064c4243e14" containerName="revoke-keys" Oct 30 21:54:45.316390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316223 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c865302a-0e53-4c41-aac6-5064c4243e14" containerName="revoke-keys" Oct 30 21:54:45.316390 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.316240 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3c5391-56f1-4ce8-b8f6-3e6603ce27ac" containerName="istio-proxy" Oct 30 21:54:45.320836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.320814 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.375774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.375748 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx"] Oct 30 21:54:45.383919 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.383895 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.384039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.383930 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.384039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.383962 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4qm\" (UniqueName: \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.425001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.424978 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:45.425104 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.424978 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vmmz2\"" Oct 30 21:54:45.484954 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.484931 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.485064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.484964 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.485064 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.484990 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4qm\" (UniqueName: \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.485622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.485602 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.487197 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.487180 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.540656 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.540632 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4qm\" (UniqueName: \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\") pod \"maas-api-revoke-sub-4bac2199-b86bx\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.634674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.634628 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:45.821070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:45.821039 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:54:46.054302 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:46.054277 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx"] Oct 30 21:54:46.055695 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:54:46.055667 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b330bf_9bb0_4865_bf01_dcce51983dc7.slice/crio-05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4 WatchSource:0}: Error finding container 05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4: Status 404 returned error can't find the container with id 05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4 Oct 30 21:54:46.539836 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:46.539800 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" event={"ID":"43b330bf-9bb0-4865-bf01-dcce51983dc7","Type":"ContainerStarted","Data":"5bd23b17bcb7ce80f38712b71c116e0853a3277e4b72cf964597c5f2011fec62"} Oct 30 21:54:46.540190 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:46.539874 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" event={"ID":"43b330bf-9bb0-4865-bf01-dcce51983dc7","Type":"ContainerStarted","Data":"05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4"} Oct 30 21:54:46.654226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:46.654141 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" podStartSLOduration=1.6541287470000001 podStartE2EDuration="1.654128747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:46.653064856 +0000 UTC m=+2600.635765275" watchObservedRunningTime="2026-10-30 21:54:46.654128747 +0000 UTC m=+2600.636829164" Oct 30 21:54:47.546269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:47.546232 4921 generic.go:356] "Generic (PLEG): container finished" podID="43b330bf-9bb0-4865-bf01-dcce51983dc7" containerID="5bd23b17bcb7ce80f38712b71c116e0853a3277e4b72cf964597c5f2011fec62" exitCode=0 Oct 30 21:54:47.546667 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:47.546325 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" event={"ID":"43b330bf-9bb0-4865-bf01-dcce51983dc7","Type":"ContainerDied","Data":"5bd23b17bcb7ce80f38712b71c116e0853a3277e4b72cf964597c5f2011fec62"} Oct 30 21:54:48.688561 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.688533 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:48.819659 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.819593 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\" (UniqueName: \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\") pod \"43b330bf-9bb0-4865-bf01-dcce51983dc7\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " Oct 30 21:54:48.819800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.819729 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\") pod \"43b330bf-9bb0-4865-bf01-dcce51983dc7\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " Oct 30 21:54:48.819800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.819778 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\") pod \"43b330bf-9bb0-4865-bf01-dcce51983dc7\" (UID: \"43b330bf-9bb0-4865-bf01-dcce51983dc7\") " Oct 30 21:54:48.820108 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.820079 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca" pod "43b330bf-9bb0-4865-bf01-dcce51983dc7" (UID: "43b330bf-9bb0-4865-bf01-dcce51983dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:48.821823 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.821799 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca" pod "43b330bf-9bb0-4865-bf01-dcce51983dc7" (UID: "43b330bf-9bb0-4865-bf01-dcce51983dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:48.821907 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.821814 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm" pod "43b330bf-9bb0-4865-bf01-dcce51983dc7" (UID: "43b330bf-9bb0-4865-bf01-dcce51983dc7"). InnerVolumeSpecName "kube-api-access-6v4qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:48.921223 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.921195 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4qm\" (UniqueName: \"kubernetes.io/projected/43b330bf-9bb0-4865-bf01-dcce51983dc7-kube-api-access-6v4qm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:48.921347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.921225 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-openshift-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:48.921347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:48.921241 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b330bf-9bb0-4865-bf01-dcce51983dc7-maas-api-xks-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:49.558118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:49.558087 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" event={"ID":"43b330bf-9bb0-4865-bf01-dcce51983dc7","Type":"ContainerDied","Data":"05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4"} Oct 30 21:54:49.558118 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:49.558114 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx" Oct 30 21:54:49.558368 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:49.558134 4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05adefb1ca011da6d601af3e0e3df55ca29868380a64a834ab7182e47ddf88d4" Oct 30 21:54:50.821800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:50.821761 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" containerID="cri-o://34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c" gracePeriod=25 Oct 30 21:54:51.967483 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:51.967449 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:53.788415 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:53.788372 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" containerID="cri-o://8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a" gracePeriod=45 Oct 30 21:54:53.836398 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:53.836357 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" containerID="cri-o://79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367" gracePeriod=45 Oct 30 21:54:54.102269 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.102230 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:54:54.138827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.138804 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:54:54.177129 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177098 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\" (UniqueName: \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.177268 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177212 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.177356 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177273 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.177435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177392 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\" (UniqueName: \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.177435 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177426 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.177555 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.177454 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\") pod \"11f14a12-67ba-46fe-a785-f30290653466\" (UID: \"11f14a12-67ba-46fe-a785-f30290653466\") " Oct 30 21:54:54.178053 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.178027 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.178386 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.178356 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.178578 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.178548 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.184347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.180982 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "kube-api-access-m8s92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:54.184347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.181863 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.184347 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.182553 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs" pod "11f14a12-67ba-46fe-a785-f30290653466" (UID: "11f14a12-67ba-46fe-a785-f30290653466"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:54.278983 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.278954 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279046 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279080 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\" (UniqueName: \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279160 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279099 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279199 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279223 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.279361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279239 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\") pod \"b8936abe-5b55-4cea-a509-7399e237916d\" (UID: \"b8936abe-5b55-4cea-a509-7399e237916d\") " Oct 30 21:54:54.279361 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279348 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.279589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279562 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.279715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279698 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.279785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279721 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f14a12-67ba-46fe-a785-f30290653466-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.279785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279737 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.279785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279749 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.279785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279762 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.279785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279774 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.280046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279786 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8s92\" (UniqueName: \"kubernetes.io/projected/11f14a12-67ba-46fe-a785-f30290653466-kube-api-access-m8s92\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.280046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279800 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.280046 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.279813 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f14a12-67ba-46fe-a785-f30290653466-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.281032 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.281009 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:54.281299 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.281274 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:54.281393 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.281346 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z" pod "b8936abe-5b55-4cea-a509-7399e237916d" (UID: "b8936abe-5b55-4cea-a509-7399e237916d"). InnerVolumeSpecName "kube-api-access-h685z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:54.380537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.380457 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8936abe-5b55-4cea-a509-7399e237916d-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.380537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.380483 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8936abe-5b55-4cea-a509-7399e237916d-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.380537 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.380493 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h685z\" (UniqueName: \"kubernetes.io/projected/b8936abe-5b55-4cea-a509-7399e237916d-kube-api-access-h685z\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:54.589239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.589207 4921 generic.go:356] "Generic (PLEG): container finished" podID="11f14a12-67ba-46fe-a785-f30290653466" containerID="8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a" exitCode=0 Oct 30 21:54:54.589442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.589252 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" Oct 30 21:54:54.589442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.589294 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" event={"ID":"11f14a12-67ba-46fe-a785-f30290653466","Type":"ContainerDied","Data":"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a"} Oct 30 21:54:54.589442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.589362 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9" event={"ID":"11f14a12-67ba-46fe-a785-f30290653466","Type":"ContainerDied","Data":"d72d4a04227129df14d54c414c84ac378bb984c607c79a2e54bf1bf557204d17"} Oct 30 21:54:54.589442 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.589380 4921 scope.go:122] "RemoveContainer" containerID="8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a" Oct 30 21:54:54.590966 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.590942 4921 generic.go:356] "Generic (PLEG): container finished" podID="b8936abe-5b55-4cea-a509-7399e237916d" containerID="79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367" exitCode=0 Oct 30 21:54:54.591084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.590980 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" event={"ID":"b8936abe-5b55-4cea-a509-7399e237916d","Type":"ContainerDied","Data":"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367"} Oct 30 21:54:54.591084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.590989 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" Oct 30 21:54:54.591084 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.590999 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll" event={"ID":"b8936abe-5b55-4cea-a509-7399e237916d","Type":"ContainerDied","Data":"22cc89b0f0d68e30113709cb96755dd0d4fe499710b148192aacd43076cc98f2"} Oct 30 21:54:54.601036 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.601014 4921 scope.go:122] "RemoveContainer" containerID="8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a" Oct 30 21:54:54.601353 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:54:54.601307 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a\": container with ID starting with 8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a not found: ID does not exist" containerID="8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a" Oct 30 21:54:54.601455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.601361 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a"} err="failed to get container status \"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a\": rpc error: code = NotFound desc = could not find container \"8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a\": container with ID starting with 8cd17efc02aa5b04dbe998d8d85655f2e16248d1de023781f96f9bc11da5408a not found: ID does not exist" Oct 30 21:54:54.601455 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.601380 4921 scope.go:122] "RemoveContainer" containerID="79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367" Oct 30 21:54:54.611845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.611822 4921 scope.go:122] "RemoveContainer" containerID="79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367" Oct 30 21:54:54.612091 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:54:54.612071 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367\": container with ID starting with 79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367 not found: ID does not exist" containerID="79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367" Oct 30 21:54:54.612155 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.612097 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367"} err="failed to get container status \"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367\": rpc error: code = NotFound desc = could not find container \"79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367\": container with ID starting with 79e09bafc657646ac289838e21160038d19a698b740a7feb6c09ef6b4c0fa367 not found: ID does not exist" Oct 30 21:54:54.754469 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.754441 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:54:54.756937 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.756912 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-w1-8ce31a-kserve-689846b5d7-wdvll"] Oct 30 21:54:54.863497 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.863470 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:54:54.869095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:54.869066 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8ce31a/distinct-2-w1-8ce31a-kserve-54649c8b5b-nhbc9"] Oct 30 21:54:56.175065 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:56.175031 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f14a12-67ba-46fe-a785-f30290653466" path="/var/lib/kubelet/pods/11f14a12-67ba-46fe-a785-f30290653466/volumes" Oct 30 21:54:56.175572 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:54:56.175556 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8936abe-5b55-4cea-a509-7399e237916d" path="/var/lib/kubelet/pods/b8936abe-5b55-4cea-a509-7399e237916d/volumes" Oct 30 21:55:01.967158 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:01.967115 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:07.539936 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:55:07.539902 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:55:11.090617 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.090564 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:55:11.228114 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.228086 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\" (UniqueName: \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\") pod \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " Oct 30 21:55:11.228278 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.228127 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\") pod \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\" (UID: \"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd\") " Oct 30 21:55:11.228519 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.228494 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume" pod "f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" (UID: "f5351234-d2ba-40e9-87a0-6f3c3f04a1dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:11.230177 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.230156 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt" pod "f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" (UID: "f5351234-d2ba-40e9-87a0-6f3c3f04a1dd"). InnerVolumeSpecName "kube-api-access-f2nmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:11.328789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.328762 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2nmt\" (UniqueName: \"kubernetes.io/projected/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-kube-api-access-f2nmt\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:11.328789 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.328788 4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd-plugins-config-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:11.717610 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.717584 4921 generic.go:356] "Generic (PLEG): container finished" podID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerID="34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c" exitCode=0 Oct 30 21:55:11.717785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.717630 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" Oct 30 21:55:11.717785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.717680 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" event={"ID":"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd","Type":"ContainerDied","Data":"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c"} Oct 30 21:55:11.717785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.717744 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr" event={"ID":"f5351234-d2ba-40e9-87a0-6f3c3f04a1dd","Type":"ContainerDied","Data":"5d442f1331902792855803bb9193c533de0c1be814eab670f9310ecdfb3fffd2"} Oct 30 21:55:11.717785 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.717775 4921 scope.go:122] "RemoveContainer" containerID="34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c" Oct 30 21:55:11.727095 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.727081 4921 scope.go:122] "RemoveContainer" containerID="34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c" Oct 30 21:55:11.727366 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:55:11.727350 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c\": container with ID starting with 34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c not found: ID does not exist" containerID="34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c" Oct 30 21:55:11.727424 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.727374 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c"} err="failed to get container status \"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c\": rpc error: code = NotFound desc = could not find container \"34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c\": container with ID starting with 34b8817fba4f4e1b129523827b7e848fef00139ad69cd2ef9e75b7001d433f3c not found: ID does not exist" Oct 30 21:55:11.880569 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.880546 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:55:11.929827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:11.929804 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8ce31a-89bd7699-d99zr"] Oct 30 21:55:12.167307 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:12.167250 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" path="/var/lib/kubelet/pods/f5351234-d2ba-40e9-87a0-6f3c3f04a1dd/volumes" Oct 30 21:55:26.410255 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410216 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410683 4921 cpu_manager.go:414] "containerMap: removing container" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410695 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410707 4921 cpu_manager.go:414] "containerMap: removing container" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410712 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410725 4921 cpu_manager.go:414] "containerMap: removing container" podUID="43b330bf-9bb0-4865-bf01-dcce51983dc7" containerName="revoke-keys" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410730 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b330bf-9bb0-4865-bf01-dcce51983dc7" containerName="revoke-keys" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410738 4921 cpu_manager.go:414] "containerMap: removing container" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410742 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410801 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5351234-d2ba-40e9-87a0-6f3c3f04a1dd" containerName="payload-processing" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410808 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f14a12-67ba-46fe-a785-f30290653466" containerName="main" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410814 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b330bf-9bb0-4865-bf01-dcce51983dc7" containerName="revoke-keys" Oct 30 21:55:26.410852 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.410820 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8936abe-5b55-4cea-a509-7399e237916d" containerName="main" Oct 30 21:55:26.413991 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.413972 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.452765 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.452727 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.452887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.452793 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5x7t\" (UniqueName: \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.452962 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.452921 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.515615 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.515589 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:55:26.515804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.515613 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:55:26.515804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.515703 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vl72m\"" Oct 30 21:55:26.515995 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.515763 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:55:26.520634 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.520603 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:55:26.553674 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.553643 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.553804 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.553787 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.553888 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.553843 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5x7t\" (UniqueName: \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.556869 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.556841 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.559306 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.559280 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.610576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.610554 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5x7t\" (UniqueName: \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\") pod \"maas-controller-6bf6cbd4cd-9strp\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:26.732060 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:26.732029 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:27.106643 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:27.106614 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:55:27.108279 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:55:27.108245 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40b1869_3e64_4c7f_9381_71659d3a7fd6.slice/crio-a16922de84a3f333dde0d33ced31620ce3851e074d597411a42576f5a6d1d536 WatchSource:0}: Error finding container a16922de84a3f333dde0d33ced31620ce3851e074d597411a42576f5a6d1d536: Status 404 returned error can't find the container with id a16922de84a3f333dde0d33ced31620ce3851e074d597411a42576f5a6d1d536 Oct 30 21:55:27.110195 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:27.110168 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:55:27.818434 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:27.818398 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" event={"ID":"a40b1869-3e64-4c7f-9381-71659d3a7fd6","Type":"ContainerStarted","Data":"a16922de84a3f333dde0d33ced31620ce3851e074d597411a42576f5a6d1d536"} Oct 30 21:55:29.832038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:29.832005 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" event={"ID":"a40b1869-3e64-4c7f-9381-71659d3a7fd6","Type":"ContainerStarted","Data":"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b"} Oct 30 21:55:29.832506 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:29.832175 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:29.950357 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:29.950296 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" podStartSLOduration=1.953926407 podStartE2EDuration="3.950284814s" totalImagesPullingTime="1.996358407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:29.947891642 +0000 UTC m=+2643.930592061" watchObservedRunningTime="2026-10-30 21:55:29.950284814 +0000 UTC m=+2643.932985244" Oct 30 21:55:31.090882 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:31.090845 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq"] Oct 30 21:55:31.093210 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:31.093190 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9cd8a4a-sbmjq"] Oct 30 21:55:32.167533 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:32.167502 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff33b735-4cad-413e-9225-95cb38cfc404" path="/var/lib/kubelet/pods/ff33b735-4cad-413e-9225-95cb38cfc404/volumes" Oct 30 21:55:37.538383 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:55:37.538285 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:55:40.839629 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:40.839601 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:55:49.147968 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.147933 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 21:55:49.153258 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.153227 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.241887 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.241864 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.241892 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.241922 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242038 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242002 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242178 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242054 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242178 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242107 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242178 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242196 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4mp\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.242257 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.242219 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.255658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.255639 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 21:55:49.343544 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343514 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343554 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343609 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343658 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343651 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343690 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343719 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343829 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343790 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343922 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4mp\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.343996 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343980 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.344100 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.343988 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.344276 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.344254 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.344426 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.344402 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.344515 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.344491 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.344570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.344550 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.346226 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.346206 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.346338 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.346286 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.401853 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.401803 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4mp\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.402508 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.402492 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6bfzs\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.469598 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.469579 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:49.893265 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.893234 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 21:55:49.894591 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:55:49.894564 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6df4698_b79b_4ef9_9f0f_4a59754b091d.slice/crio-becebced3ee1b978b5f53ec50930b0d756a9cf8874709b1b723c6bc8363edd1d WatchSource:0}: Error finding container becebced3ee1b978b5f53ec50930b0d756a9cf8874709b1b723c6bc8363edd1d: Status 404 returned error can't find the container with id becebced3ee1b978b5f53ec50930b0d756a9cf8874709b1b723c6bc8363edd1d Oct 30 21:55:49.896725 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.896693 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:49.896813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.896753 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:49.896813 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.896782 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:49.942023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:49.942000 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" event={"ID":"d6df4698-b79b-4ef9-9f0f-4a59754b091d","Type":"ContainerStarted","Data":"becebced3ee1b978b5f53ec50930b0d756a9cf8874709b1b723c6bc8363edd1d"} Oct 30 21:55:50.949502 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:50.949467 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" event={"ID":"d6df4698-b79b-4ef9-9f0f-4a59754b091d","Type":"ContainerStarted","Data":"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8"} Oct 30 21:55:51.470011 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:51.469981 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:51.471576 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:51.471552 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:51.471696 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:51.471598 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:52.470491 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:52.470457 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:52.470922 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:52.470507 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:53.469873 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:53.469835 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:53.470070 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:53.469888 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:54.470392 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:54.470355 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:54.470783 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:54.470412 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:55.469949 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:55.469915 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:55.470117 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:55.469968 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:56.470371 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:56.470338 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:55:56.470808 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:56.470392 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 21:55:58.471586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:58.471550 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:58.472014 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:58.471603 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:59.512449 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.512414 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:59.513012 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.512471 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:59.764099 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.764017 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:59.764461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.764435 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:59.765182 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.765165 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 21:55:59.934023 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:55:59.933980 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podStartSLOduration=10.933968098 podStartE2EDuration="10.933968098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:51.119952738 +0000 UTC m=+2665.102653155" watchObservedRunningTime="2026-10-30 21:55:59.933968098 +0000 UTC m=+2673.916668518" Oct 30 21:56:02.520903 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.520869 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:56:02.521290 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.521035 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" podUID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" containerName="manager" containerID="cri-o://f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b" gracePeriod=30 Oct 30 21:56:02.772251 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.772200 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:56:02.886300 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.886274 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\") pod \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " Oct 30 21:56:02.886456 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.886364 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\") pod \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " Oct 30 21:56:02.886456 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.886412 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\" (UniqueName: \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\") pod \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\" (UID: \"a40b1869-3e64-4c7f-9381-71659d3a7fd6\") " Oct 30 21:56:02.888234 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.888202 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert" pod "a40b1869-3e64-4c7f-9381-71659d3a7fd6" (UID: "a40b1869-3e64-4c7f-9381-71659d3a7fd6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:02.888352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.888240 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs" pod "a40b1869-3e64-4c7f-9381-71659d3a7fd6" (UID: "a40b1869-3e64-4c7f-9381-71659d3a7fd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:02.888352 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.888304 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t" pod "a40b1869-3e64-4c7f-9381-71659d3a7fd6" (UID: "a40b1869-3e64-4c7f-9381-71659d3a7fd6"). InnerVolumeSpecName "kube-api-access-j5x7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:02.987589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.987550 4921 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-webhook-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:02.987589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.987579 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a40b1869-3e64-4c7f-9381-71659d3a7fd6-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:02.987589 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:02.987593 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5x7t\" (UniqueName: \"kubernetes.io/projected/a40b1869-3e64-4c7f-9381-71659d3a7fd6-kube-api-access-j5x7t\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:03.031597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.031529 4921 generic.go:356] "Generic (PLEG): container finished" podID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" containerID="f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b" exitCode=0 Oct 30 21:56:03.031597 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.031575 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" Oct 30 21:56:03.031758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.031607 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" event={"ID":"a40b1869-3e64-4c7f-9381-71659d3a7fd6","Type":"ContainerDied","Data":"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b"} Oct 30 21:56:03.031758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.031656 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-9strp" event={"ID":"a40b1869-3e64-4c7f-9381-71659d3a7fd6","Type":"ContainerDied","Data":"a16922de84a3f333dde0d33ced31620ce3851e074d597411a42576f5a6d1d536"} Oct 30 21:56:03.031758 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.031678 4921 scope.go:122] "RemoveContainer" containerID="f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b" Oct 30 21:56:03.041532 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.041515 4921 scope.go:122] "RemoveContainer" containerID="f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b" Oct 30 21:56:03.041775 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:56:03.041759 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b\": container with ID starting with f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b not found: ID does not exist" containerID="f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b" Oct 30 21:56:03.041828 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.041782 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b"} err="failed to get container status \"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b\": rpc error: code = NotFound desc = could not find container \"f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b\": container with ID starting with f566c39ff50fcad2dd2e17da1e43faffadab9b636ed2b29d4c368192af933c5b not found: ID does not exist" Oct 30 21:56:03.196007 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.195987 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:56:03.199450 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:03.199430 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-9strp"] Oct 30 21:56:04.166817 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:04.166784 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" path="/var/lib/kubelet/pods/a40b1869-3e64-4c7f-9381-71659d3a7fd6/volumes" Oct 30 21:56:07.538801 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:56:07.538765 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:56:14.015501 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.015469 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-tbp75"] Oct 30 21:56:14.016090 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.016070 4921 cpu_manager.go:414] "containerMap: removing container" podUID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" containerName="manager" Oct 30 21:56:14.016178 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.016093 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" containerName="manager" Oct 30 21:56:14.016243 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.016230 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b1869-3e64-4c7f-9381-71659d3a7fd6" containerName="manager" Oct 30 21:56:14.020735 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.020714 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.072715 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.072693 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:56:14.082444 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.082413 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx4xc\" (UniqueName: \"kubernetes.io/projected/e26644f3-d7bb-4c37-8c13-a4aacb616d72-kube-api-access-hx4xc\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.082570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.082453 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.082570 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.082559 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.122039 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.122019 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:56:14.122264 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.122239 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vl72m\"" Oct 30 21:56:14.122389 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.122311 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:56:14.126564 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.126540 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-tbp75"] Oct 30 21:56:14.183471 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.183448 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.183602 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.183590 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx4xc\" (UniqueName: \"kubernetes.io/projected/e26644f3-d7bb-4c37-8c13-a4aacb616d72-kube-api-access-hx4xc\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.183645 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.183616 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.185781 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.185763 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-webhook-cert\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.185939 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.185907 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26644f3-d7bb-4c37-8c13-a4aacb616d72-metrics-certs\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.243767 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.243748 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx4xc\" (UniqueName: \"kubernetes.io/projected/e26644f3-d7bb-4c37-8c13-a4aacb616d72-kube-api-access-hx4xc\") pod \"maas-controller-6bf6cbd4cd-tbp75\" (UID: \"e26644f3-d7bb-4c37-8c13-a4aacb616d72\") " pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.333458 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.333410 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:14.708297 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:14.708269 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6bf6cbd4cd-tbp75"] Oct 30 21:56:14.708696 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:56:14.708670 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26644f3_d7bb_4c37_8c13_a4aacb616d72.slice/crio-42179151be7359a7c3b0eb5dfdf68bcb1f04398f26834c4126c86934b2b409cd WatchSource:0}: Error finding container 42179151be7359a7c3b0eb5dfdf68bcb1f04398f26834c4126c86934b2b409cd: Status 404 returned error can't find the container with id 42179151be7359a7c3b0eb5dfdf68bcb1f04398f26834c4126c86934b2b409cd Oct 30 21:56:15.102526 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:15.102463 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" event={"ID":"e26644f3-d7bb-4c37-8c13-a4aacb616d72","Type":"ContainerStarted","Data":"42179151be7359a7c3b0eb5dfdf68bcb1f04398f26834c4126c86934b2b409cd"} Oct 30 21:56:16.110650 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:16.110615 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" event={"ID":"e26644f3-d7bb-4c37-8c13-a4aacb616d72","Type":"ContainerStarted","Data":"1837d0285162028760fbd7a9cf92d4a8758023a3a8cb2e06ca39a80530d77770"} Oct 30 21:56:16.111069 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:16.110716 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:16.229251 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:16.229203 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" podStartSLOduration=2.375377775 podStartE2EDuration="3.229191788s" totalImagesPullingTime="853.814013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:16.227592334 +0000 UTC m=+2690.210292753" watchObservedRunningTime="2026-10-30 21:56:16.229191788 +0000 UTC m=+2690.211892244" Oct 30 21:56:26.847657 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:26.847628 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:56:26.857994 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:26.857975 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:56:26.900906 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:26.900888 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 21:56:26.911586 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:26.911571 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 21:56:27.118849 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:27.118789 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6bf6cbd4cd-tbp75" Oct 30 21:56:27.603845 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:56:27.603810 4921 scope.go:122] "RemoveContainer" containerID="7e27ab0d1c4f5171c644435eb2c9bf78c5a320e16f5202de4455f6e472b938d6" Oct 30 21:56:37.538413 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:56:37.538377 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:57:07.539263 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:57:07.539149 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:57:19.875977 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.875944 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 21:57:19.879827 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.879804 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.933571 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.933538 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.933835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.933816 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8rt\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.933974 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.933959 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934142 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934119 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934187 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934224 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934275 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934249 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934273 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:19.934505 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:19.934302 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.035680 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.035648 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.036350 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.036310 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.036841 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.036823 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.036987 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.036969 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.037113 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.037098 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.037239 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.037223 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.037461 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.037441 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8rt\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.039611 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.039593 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.040093 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.040076 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.040622 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.038366 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.040768 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.039300 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.040875 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.040008 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.041047 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.036760 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.041149 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.036235 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.057401 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.057368 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.062499 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.062483 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.085134 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.085112 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 21:57:20.098612 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.098583 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8rt\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.113077 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.113054 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-qm6kg\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.192245 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.192178 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:20.637061 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.637036 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 21:57:20.638291 ip-10-0-137-220 kubenswrapper[4921]: W1030 21:57:20.638265 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bcd4b4_0868_47eb_bcc7_3fe245974842.slice/crio-bd7d08d1470994364b64d025bf7935f790542bba5bfc2a58136e8e6946acff36 WatchSource:0}: Error finding container bd7d08d1470994364b64d025bf7935f790542bba5bfc2a58136e8e6946acff36: Status 404 returned error can't find the container with id bd7d08d1470994364b64d025bf7935f790542bba5bfc2a58136e8e6946acff36 Oct 30 21:57:20.640254 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.640220 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:20.640373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.640295 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:20.640373 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:20.640355 4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:21.479382 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:21.479349 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" event={"ID":"f5bcd4b4-0868-47eb-bcc7-3fe245974842","Type":"ContainerStarted","Data":"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974"} Oct 30 21:57:21.479800 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:21.479403 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" event={"ID":"f5bcd4b4-0868-47eb-bcc7-3fe245974842","Type":"ContainerStarted","Data":"bd7d08d1470994364b64d025bf7935f790542bba5bfc2a58136e8e6946acff36"} Oct 30 21:57:21.702342 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:21.698421 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podStartSLOduration=2.698403852 podStartE2EDuration="2.698403852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:21.646827614 +0000 UTC m=+2755.629528031" watchObservedRunningTime="2026-10-30 21:57:21.698403852 +0000 UTC m=+2755.681104269" Oct 30 21:57:22.193133 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:22.193102 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:22.194711 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:22.194686 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:22.194835 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:22.194728 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:23.192642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:23.192604 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:23.193110 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:23.192665 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:24.192642 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:24.192605 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:24.192809 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:24.192657 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:25.192666 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:25.192634 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:25.192870 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:25.192699 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:26.193001 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:26.192972 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:26.193388 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:26.193025 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:27.193073 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:27.193035 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 21:57:27.193468 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:27.193095 4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 21:57:28.642143 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:28.642110 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:28.643298 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:28.643274 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:28.643906 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:57:28.643888 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 21:57:37.538081 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:57:37.538045 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:58:00.818339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:58:00.811404 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:58:00.818339 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:58:00.811466 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:07.538618 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:58:07.538580 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:58:37.538774 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:58:37.538699 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:59:07.539212 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:59:07.539172 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:59:37.539717 ip-10-0-137-220 kubenswrapper[4921]: E1030 21:59:37.539679 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 21:59:49.069774 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:59:49.069743 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx"] Oct 30 21:59:49.119000 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:59:49.118973 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac2199-b86bx"] Oct 30 21:59:50.167414 ip-10-0-137-220 kubenswrapper[4921]: I1030 21:59:50.167377 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b330bf-9bb0-4865-bf01-dcce51983dc7" path="/var/lib/kubelet/pods/43b330bf-9bb0-4865-bf01-dcce51983dc7/volumes" Oct 30 22:00:07.539466 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:00:07.539387 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:00:27.656881 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:00:27.656845 4921 scope.go:122] "RemoveContainer" containerID="e9687abd15f7168f747100ff057734ad6bdfb1f7a803dc64638b635ba34e1cf7" Oct 30 22:00:37.539640 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:00:37.539593 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:01:07.538233 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:01:07.538195 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:01:26.888091 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:26.888061 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:01:26.899220 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:26.899200 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:01:26.942287 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:26.942268 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:01:26.953853 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:26.953838 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:01:27.686132 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:27.686102 4921 scope.go:122] "RemoveContainer" containerID="5bd23b17bcb7ce80f38712b71c116e0853a3277e4b72cf964597c5f2011fec62" Oct 30 22:01:35.893469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:35.893432 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:01:35.893873 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:01:35.893507 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:01:37.539005 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:01:37.538924 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:02:07.540605 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:02:07.540492 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:02:37.540938 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:02:37.540900 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:03:07.541889 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:03:07.541803 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:03:30.820342 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:30.819713 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6bfzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:30.820342 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:30.819791 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.541983 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:03:37.541936 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:03:56.967770 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:56.967727 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:03:56.971577 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:56.971550 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.075983 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.075946 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wqlw4\"" Oct 30 22:03:57.076191 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.076115 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-197a40-serving-cert\"" Oct 30 22:03:57.076191 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.076113 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:03:57.079417 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.079390 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:03:57.083755 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.083733 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvh8\" (UniqueName: \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.083818 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.083796 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.083966 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.083939 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.084065 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.084050 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.185500 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.185460 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.185773 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.185542 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.185773 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.185585 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvh8\" (UniqueName: \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.185773 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:03:57.185613 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-serving-cert: secret "maas-api-e2e-worker-single-main-197a40-serving-cert" not found Oct 30 22:03:57.185773 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.185633 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.185773 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:03:57.185698 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls podName:0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:57.685680915 +0000 UTC m=+3151.668381317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls") pod "maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" (UID: "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8") : secret "maas-api-e2e-worker-single-main-197a40-serving-cert" not found Oct 30 22:03:57.186404 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.186375 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.188150 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.188128 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.242752 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.242677 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvh8\" (UniqueName: \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.691255 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.691155 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.693891 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.693861 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:57.885821 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:57.885778 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:03:58.263164 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:58.263127 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:03:58.264931 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:03:58.264901 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c90e9ad_f7de_40e3_b7e8_ef26e2e5f4e8.slice/crio-cf34524282ba0a12b2eb153d1d6956340eb4bdabb9c4e98a7cc85a1bc33b6bf3 WatchSource:0}: Error finding container cf34524282ba0a12b2eb153d1d6956340eb4bdabb9c4e98a7cc85a1bc33b6bf3: Status 404 returned error can't find the container with id cf34524282ba0a12b2eb153d1d6956340eb4bdabb9c4e98a7cc85a1bc33b6bf3 Oct 30 22:03:58.266464 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:58.266400 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:03:59.069133 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:03:59.069097 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" event={"ID":"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8","Type":"ContainerStarted","Data":"cf34524282ba0a12b2eb153d1d6956340eb4bdabb9c4e98a7cc85a1bc33b6bf3"} Oct 30 22:04:00.076340 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:00.076274 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" event={"ID":"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8","Type":"ContainerStarted","Data":"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f"} Oct 30 22:04:00.076804 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:00.076383 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:04:00.196184 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:00.196124 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" podStartSLOduration=3.303433178 podStartE2EDuration="4.196110066s" totalImagesPullingTime="892.676888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:00.193635569 +0000 UTC m=+3154.176335988" watchObservedRunningTime="2026-10-30 22:04:00.196110066 +0000 UTC m=+3154.178810486" Oct 30 22:04:06.088636 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:06.088600 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:04:07.548756 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:04:07.548714 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:04:18.668230 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.668187 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:04:18.672162 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.672142 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706500 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706464 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706500 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706505 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhj9\" (UniqueName: \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706789 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706530 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706789 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706727 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706908 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706789 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.706908 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.706829 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.773348 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.773281 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-197a40\"/\"openshift-service-ca.crt\"" Oct 30 22:04:18.773348 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.773304 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-197a40\"/\"default-dockercfg-gh4tl\"" Oct 30 22:04:18.773348 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.773281 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-197a40\"/\"kube-root-ca.crt\"" Oct 30 22:04:18.773685 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.773475 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-197a40\"/\"simulator-single-main-197a40-kserve-self-signed-certs\"" Oct 30 22:04:18.776519 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.776487 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:04:18.809232 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809195 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809480 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809255 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809480 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809372 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809480 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809399 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhj9\" (UniqueName: \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809480 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809432 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809722 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809563 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809722 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809696 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.809941 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.809917 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.810397 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.810369 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.812978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.812956 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.813174 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.813156 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.869376 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.869340 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhj9\" (UniqueName: \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\") pod \"simulator-single-main-197a40-kserve-864887cdb4-5cvzk\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:18.984827 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:18.984792 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:19.375948 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:19.375914 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:04:19.377533 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:04:19.377506 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7cd43e6_55a9_4d9f_98ad_3aa2e1d3ea2e.slice/crio-547b061b9f2933836d9b39d633ceaa225144a992e3d46e10b74d22641c38f6c9 WatchSource:0}: Error finding container 547b061b9f2933836d9b39d633ceaa225144a992e3d46e10b74d22641c38f6c9: Status 404 returned error can't find the container with id 547b061b9f2933836d9b39d633ceaa225144a992e3d46e10b74d22641c38f6c9 Oct 30 22:04:20.204518 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:20.204485 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" event={"ID":"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e","Type":"ContainerStarted","Data":"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47"} Oct 30 22:04:20.204953 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:20.204529 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" event={"ID":"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e","Type":"ContainerStarted","Data":"547b061b9f2933836d9b39d633ceaa225144a992e3d46e10b74d22641c38f6c9"} Oct 30 22:04:20.320962 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:20.320890 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" podStartSLOduration=2.320873048 podStartE2EDuration="2.320873048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:20.31856202 +0000 UTC m=+3174.301262441" watchObservedRunningTime="2026-10-30 22:04:20.320873048 +0000 UTC m=+3174.303573466" Oct 30 22:04:20.985543 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:20.985511 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:28.985420 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:28.985377 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:28.998252 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:28.998222 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:29.271350 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:04:29.271252 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:04:37.540598 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:04:37.540504 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:05:07.540156 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:05:07.540121 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:05:20.338939 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:20.338902 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:05:34.176820 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:34.176770 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:05:34.181084 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:34.181048 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" podUID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" containerName="maas-api" containerID="cri-o://25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f" gracePeriod=30 Oct 30 22:05:35.360218 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.360155 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" podUID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" containerName="main" containerID="cri-o://3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47" gracePeriod=45 Oct 30 22:05:35.490484 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.490460 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:05:35.556656 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.556625 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\") pod \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " Oct 30 22:05:35.556820 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.556682 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\" (UniqueName: \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\") pod \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " Oct 30 22:05:35.556820 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.556760 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\") pod \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " Oct 30 22:05:35.556820 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.556801 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") pod \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\" (UID: \"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8\") " Oct 30 22:05:35.557371 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.557034 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca" pod "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" (UID: "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:35.557371 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.557308 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.559839 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.559812 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls" pod "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" (UID: "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:35.559944 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.559841 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs" pod "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" (UID: "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:35.559944 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.559850 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8" pod "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" (UID: "0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8"). InnerVolumeSpecName "kube-api-access-drvh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:35.602379 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.602358 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:05:35.658574 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658546 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658597 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658634 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658655 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658880 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658715 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658880 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658760 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\" (UniqueName: \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\") pod \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\" (UID: \"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e\") " Oct 30 22:05:35.658880 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658806 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.659042 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658930 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.659042 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.658947 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659078 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drvh8\" (UniqueName: \"kubernetes.io/projected/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-kube-api-access-drvh8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659100 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659117 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659132 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659146 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.659161 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.659160 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.660776 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.660757 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.660869 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.660851 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:35.660907 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.660860 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9" pod "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" (UID: "e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e"). InnerVolumeSpecName "kube-api-access-kvhj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:35.675464 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.675438 4921 generic.go:356] "Generic (PLEG): container finished" podID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" containerID="3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47" exitCode=0 Oct 30 22:05:35.675562 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.675482 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" Oct 30 22:05:35.675562 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.675522 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" event={"ID":"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e","Type":"ContainerDied","Data":"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47"} Oct 30 22:05:35.675661 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.675564 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk" event={"ID":"e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e","Type":"ContainerDied","Data":"547b061b9f2933836d9b39d633ceaa225144a992e3d46e10b74d22641c38f6c9"} Oct 30 22:05:35.675661 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.675584 4921 scope.go:122] "RemoveContainer" containerID="3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47" Oct 30 22:05:35.677048 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.677029 4921 generic.go:356] "Generic (PLEG): container finished" podID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" containerID="25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f" exitCode=0 Oct 30 22:05:35.677156 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.677072 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" Oct 30 22:05:35.677156 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.677086 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" event={"ID":"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8","Type":"ContainerDied","Data":"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f"} Oct 30 22:05:35.677156 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.677120 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp" event={"ID":"0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8","Type":"ContainerDied","Data":"cf34524282ba0a12b2eb153d1d6956340eb4bdabb9c4e98a7cc85a1bc33b6bf3"} Oct 30 22:05:35.685413 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.685396 4921 scope.go:122] "RemoveContainer" containerID="3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47" Oct 30 22:05:35.685683 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:05:35.685665 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47\": container with ID starting with 3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47 not found: ID does not exist" containerID="3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47" Oct 30 22:05:35.685745 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.685690 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47"} err="failed to get container status \"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47\": rpc error: code = NotFound desc = could not find container \"3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47\": container with ID starting with 3b0059ed2a8cf4c46d28eb285de519830c583d56b0f100957efae2d136a2da47 not found: ID does not exist" Oct 30 22:05:35.685745 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.685706 4921 scope.go:122] "RemoveContainer" containerID="25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f" Oct 30 22:05:35.693995 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.693973 4921 scope.go:122] "RemoveContainer" containerID="25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f" Oct 30 22:05:35.694224 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:05:35.694207 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f\": container with ID starting with 25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f not found: ID does not exist" containerID="25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f" Oct 30 22:05:35.694273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.694232 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f"} err="failed to get container status \"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f\": rpc error: code = NotFound desc = could not find container \"25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f\": container with ID starting with 25145a8767fa2f2cd39276457b7ccf3f4476509a9ebbcb16143540e3e7507f4f not found: ID does not exist" Oct 30 22:05:35.760284 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.760243 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.760284 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.760283 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhj9\" (UniqueName: \"kubernetes.io/projected/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-kube-api-access-kvhj9\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.760578 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.760294 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.830000 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.829896 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:05:35.832653 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.832625 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-197a40-58898ccb84-dn7wp"] Oct 30 22:05:35.939788 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.939756 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:05:35.942277 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:35.942251 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-197a40/simulator-single-main-197a40-kserve-864887cdb4-5cvzk"] Oct 30 22:05:36.167470 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:36.167391 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" path="/var/lib/kubelet/pods/0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8/volumes" Oct 30 22:05:36.167817 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:05:36.167803 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" path="/var/lib/kubelet/pods/e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e/volumes" Oct 30 22:05:37.539999 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:05:37.539964 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:06:07.539882 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:06:07.539794 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:06:26.927877 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:26.927847 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:06:26.947697 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:26.947669 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:06:26.990768 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:26.990735 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:06:27.007080 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:27.007060 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:06:37.539289 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:06:37.539252 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:06:53.271750 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.271715 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 22:06:53.272187 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.271913 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" podUID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" containerName="manager" containerID="cri-o://7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf" gracePeriod=10 Oct 30 22:06:53.539968 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.539943 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 22:06:53.711745 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.711722 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\" (UniqueName: \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\") pod \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " Oct 30 22:06:53.711875 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.711774 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\") pod \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\" (UID: \"abb6e432-ebec-4cfa-bb6b-39badd83bb4e\") " Oct 30 22:06:53.712238 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.712206 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume" pod "abb6e432-ebec-4cfa-bb6b-39badd83bb4e" (UID: "abb6e432-ebec-4cfa-bb6b-39badd83bb4e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:53.713839 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.713811 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8" pod "abb6e432-ebec-4cfa-bb6b-39badd83bb4e" (UID: "abb6e432-ebec-4cfa-bb6b-39badd83bb4e"). InnerVolumeSpecName "kube-api-access-rnln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:53.813139 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.813068 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnln8\" (UniqueName: \"kubernetes.io/projected/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-kube-api-access-rnln8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:53.813139 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:53.813097 4921 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb6e432-ebec-4cfa-bb6b-39badd83bb4e-extensions-socket-volume\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:06:54.173828 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.173755 4921 generic.go:356] "Generic (PLEG): container finished" podID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" containerID="7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf" exitCode=0 Oct 30 22:06:54.173828 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.173801 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" Oct 30 22:06:54.174010 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.173832 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" event={"ID":"abb6e432-ebec-4cfa-bb6b-39badd83bb4e","Type":"ContainerDied","Data":"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf"} Oct 30 22:06:54.174010 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.173885 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq" event={"ID":"abb6e432-ebec-4cfa-bb6b-39badd83bb4e","Type":"ContainerDied","Data":"d1b6ca54cb8092b6bcab8c2750348f88c37ab9283b6879b16fa907ba97b3cd20"} Oct 30 22:06:54.174010 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.173913 4921 scope.go:122] "RemoveContainer" containerID="7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf" Oct 30 22:06:54.186094 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.186081 4921 scope.go:122] "RemoveContainer" containerID="7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf" Oct 30 22:06:54.186365 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:06:54.186341 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf\": container with ID starting with 7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf not found: ID does not exist" containerID="7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf" Oct 30 22:06:54.186448 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.186376 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf"} err="failed to get container status \"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf\": rpc error: code = NotFound desc = could not find container \"7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf\": container with ID starting with 7e06044b3c52bfb286dd964448ca1cddf0b649a06be882115abf5b5f978f6edf not found: ID does not exist" Oct 30 22:06:54.385611 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.385577 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 22:06:54.388950 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:54.388929 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wvcpq"] Oct 30 22:06:56.167074 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:06:56.167041 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" path="/var/lib/kubelet/pods/abb6e432-ebec-4cfa-bb6b-39badd83bb4e/volumes" Oct 30 22:07:07.539671 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:07:07.539635 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:07:37.540352 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:07:37.540256 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:07:56.944911 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.944873 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh"] Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945308 4921 cpu_manager.go:414] "containerMap: removing container" podUID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" containerName="main" Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945337 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" containerName="main" Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945356 4921 cpu_manager.go:414] "containerMap: removing container" podUID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" containerName="maas-api" Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945361 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" containerName="maas-api" Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945379 4921 cpu_manager.go:414] "containerMap: removing container" podUID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" containerName="manager" Oct 30 22:07:56.945435 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945385 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" containerName="manager" Oct 30 22:07:56.945638 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945466 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cd43e6-55a9-4d9f-98ad-3aa2e1d3ea2e" containerName="main" Oct 30 22:07:56.945638 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945475 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c90e9ad-f7de-40e3-b7e8-ef26e2e5f4e8" containerName="maas-api" Oct 30 22:07:56.945638 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.945485 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb6e432-ebec-4cfa-bb6b-39badd83bb4e" containerName="manager" Oct 30 22:07:56.948708 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:56.948688 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.004907 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.004868 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh"] Oct 30 22:07:57.036521 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.036487 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjft\" (UniqueName: \"kubernetes.io/projected/6fa66274-3adf-4103-a622-04048b8732d3-kube-api-access-8wjft\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.036731 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.036535 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fa66274-3adf-4103-a622-04048b8732d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.051246 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.051216 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bkx8t\"" Oct 30 22:07:57.137205 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.137167 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjft\" (UniqueName: \"kubernetes.io/projected/6fa66274-3adf-4103-a622-04048b8732d3-kube-api-access-8wjft\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.137415 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.137211 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fa66274-3adf-4103-a622-04048b8732d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.137639 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.137619 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fa66274-3adf-4103-a622-04048b8732d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.193829 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.193796 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjft\" (UniqueName: \"kubernetes.io/projected/6fa66274-3adf-4103-a622-04048b8732d3-kube-api-access-8wjft\") pod \"kuadrant-operator-controller-manager-678d87766f-czknh\" (UID: \"6fa66274-3adf-4103-a622-04048b8732d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.262780 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.262745 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:07:57.741769 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:57.741739 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh"] Oct 30 22:07:57.743500 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:07:57.743475 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa66274_3adf_4103_a622_04048b8732d3.slice/crio-db99234bab736aa2214550d71462f3e53f6d321f03951c55727a322633ef2d21 WatchSource:0}: Error finding container db99234bab736aa2214550d71462f3e53f6d321f03951c55727a322633ef2d21: Status 404 returned error can't find the container with id db99234bab736aa2214550d71462f3e53f6d321f03951c55727a322633ef2d21 Oct 30 22:07:58.533480 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:58.533446 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" event={"ID":"6fa66274-3adf-4103-a622-04048b8732d3","Type":"ContainerStarted","Data":"7accf50df7e167c8a2d5328f328245037af972420c4aaa5a50d305076733bfe4"} Oct 30 22:07:58.533988 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:58.533504 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" event={"ID":"6fa66274-3adf-4103-a622-04048b8732d3","Type":"ContainerStarted","Data":"db99234bab736aa2214550d71462f3e53f6d321f03951c55727a322633ef2d21"} Oct 30 22:07:58.533988 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:07:58.533593 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:08:07.539397 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:08:07.539359 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:08:09.544703 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:08:09.544669 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" Oct 30 22:08:09.767334 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:08:09.767254 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-czknh" podStartSLOduration=13.767232961 podStartE2EDuration="13.767232961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:58.770003498 +0000 UTC m=+3392.752703917" watchObservedRunningTime="2026-10-30 22:08:09.767232961 +0000 UTC m=+3403.749933381" Oct 30 22:08:20.913342 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:08:20.906443 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:20.913342 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:08:20.906519 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:37.540034 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:08:37.539995 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:09:07.539392 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:09:07.539290 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:09:30.662562 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.662525 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:09:30.666493 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.666474 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.704017 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.703988 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.704203 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.704108 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.704203 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.704175 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.704310 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.704297 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d59f\" (UniqueName: \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.767473 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.767437 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:09:30.767473 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.767442 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dc7946-serving-cert\"" Oct 30 22:09:30.767761 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.767501 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wqlw4\"" Oct 30 22:09:30.770447 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.770423 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:09:30.805453 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.805414 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.805652 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.805472 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.805652 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.805543 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d59f\" (UniqueName: \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.805652 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.805568 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.805652 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:09:30.805584 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-serving-cert: secret "maas-api-e2e-worker-single-main-dc7946-serving-cert" not found Oct 30 22:09:30.805900 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:09:30.805670 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls podName:be6a3311-b2a1-4a49-919b-564b2c019efd nodeName:}" failed. No retries permitted until 2026-10-30 22:09:31.305648529 +0000 UTC m=+3485.288348926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls") pod "maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" (UID: "be6a3311-b2a1-4a49-919b-564b2c019efd") : secret "maas-api-e2e-worker-single-main-dc7946-serving-cert" not found Oct 30 22:09:30.806198 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.806170 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.808046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.808023 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:30.862680 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:30.862650 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d59f\" (UniqueName: \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:31.311683 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:31.311641 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:31.314242 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:31.314218 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:31.579618 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:31.579529 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:31.955919 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:31.955891 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:09:31.962963 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:09:31.958662 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6a3311_b2a1_4a49_919b_564b2c019efd.slice/crio-bf50b2f4bcb02a4b5aa2162e90c34fb6fda23bffba58c5fe9dae2061caa76df9 WatchSource:0}: Error finding container bf50b2f4bcb02a4b5aa2162e90c34fb6fda23bffba58c5fe9dae2061caa76df9: Status 404 returned error can't find the container with id bf50b2f4bcb02a4b5aa2162e90c34fb6fda23bffba58c5fe9dae2061caa76df9 Oct 30 22:09:31.963988 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:31.963967 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:09:32.121678 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:32.121637 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" event={"ID":"be6a3311-b2a1-4a49-919b-564b2c019efd","Type":"ContainerStarted","Data":"bf50b2f4bcb02a4b5aa2162e90c34fb6fda23bffba58c5fe9dae2061caa76df9"} Oct 30 22:09:33.130693 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:33.130591 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" event={"ID":"be6a3311-b2a1-4a49-919b-564b2c019efd","Type":"ContainerStarted","Data":"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94"} Oct 30 22:09:33.131195 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:33.130723 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:09:33.247214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:33.247149 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" podStartSLOduration=2.432625333 podStartE2EDuration="3.247135296s" totalImagesPullingTime="814.509963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:33.245123084 +0000 UTC m=+3487.227823514" watchObservedRunningTime="2026-10-30 22:09:33.247135296 +0000 UTC m=+3487.229835714" Oct 30 22:09:37.540389 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:09:37.540349 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:09:39.142374 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:09:39.142336 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:10:04.769148 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.769103 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:10:04.773193 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.773169 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.827197 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.827167 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dc7946\"/\"openshift-service-ca.crt\"" Oct 30 22:10:04.867156 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867116 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp42\" (UniqueName: \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.867353 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867215 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.867353 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867254 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.867353 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867290 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.867353 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867307 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.867551 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.867411 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.876550 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.876519 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dc7946\"/\"default-dockercfg-p8pqr\"" Oct 30 22:10:04.876691 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.876551 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dc7946\"/\"simulator-single-main-dc7946-kserve-self-signed-certs\"" Oct 30 22:10:04.879587 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.879564 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:10:04.929139 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.929109 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dc7946\"/\"kube-root-ca.crt\"" Oct 30 22:10:04.967958 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.967918 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp42\" (UniqueName: \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.967983 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968021 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968061 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968087 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968121 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968613 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968583 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968747 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968628 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.968747 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.968665 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.970636 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.970605 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:04.970757 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:04.970693 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:05.024203 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:05.024114 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp42\" (UniqueName: \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\") pod \"simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:05.085703 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:05.085661 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:05.462700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:05.462670 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:10:05.464483 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:10:05.464452 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1543c9ee_5fd7_427a_871a_776c58ad5420.slice/crio-a4069fd6960ad10163610e63a76d384d0cd5d365ba03f6a0f43ed9fa67f450af WatchSource:0}: Error finding container a4069fd6960ad10163610e63a76d384d0cd5d365ba03f6a0f43ed9fa67f450af: Status 404 returned error can't find the container with id a4069fd6960ad10163610e63a76d384d0cd5d365ba03f6a0f43ed9fa67f450af Oct 30 22:10:06.326775 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:06.326733 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" event={"ID":"1543c9ee-5fd7-427a-871a-776c58ad5420","Type":"ContainerStarted","Data":"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5"} Oct 30 22:10:06.327193 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:06.326785 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" event={"ID":"1543c9ee-5fd7-427a-871a-776c58ad5420","Type":"ContainerStarted","Data":"a4069fd6960ad10163610e63a76d384d0cd5d365ba03f6a0f43ed9fa67f450af"} Oct 30 22:10:06.442482 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:06.442427 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" podStartSLOduration=2.44241278 podStartE2EDuration="2.44241278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:06.440837237 +0000 UTC m=+3520.423537656" watchObservedRunningTime="2026-10-30 22:10:06.44241278 +0000 UTC m=+3520.425113198" Oct 30 22:10:07.086112 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:07.086061 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:07.541468 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:10:07.541434 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:10:15.087101 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:15.087064 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:15.099458 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:15.099426 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:15.405150 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:10:15.405052 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:10:37.548332 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:10:37.548233 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:11:07.543756 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:11:07.543713 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:11:10.394281 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:10.394240 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:11:17.379439 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:17.379406 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:11:17.379852 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:17.379642 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" podUID="be6a3311-b2a1-4a49-919b-564b2c019efd" containerName="maas-api" containerID="cri-o://4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94" gracePeriod=30 Oct 30 22:11:18.644632 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.644600 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:11:18.653940 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.653914 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") pod \"be6a3311-b2a1-4a49-919b-564b2c019efd\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " Oct 30 22:11:18.654073 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.653978 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\") pod \"be6a3311-b2a1-4a49-919b-564b2c019efd\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " Oct 30 22:11:18.654146 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.654074 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\" (UniqueName: \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\") pod \"be6a3311-b2a1-4a49-919b-564b2c019efd\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " Oct 30 22:11:18.655916 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.655890 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs" pod "be6a3311-b2a1-4a49-919b-564b2c019efd" (UID: "be6a3311-b2a1-4a49-919b-564b2c019efd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:18.656013 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.655963 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f" pod "be6a3311-b2a1-4a49-919b-564b2c019efd" (UID: "be6a3311-b2a1-4a49-919b-564b2c019efd"). InnerVolumeSpecName "kube-api-access-9d59f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:18.656054 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.656011 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls" pod "be6a3311-b2a1-4a49-919b-564b2c019efd" (UID: "be6a3311-b2a1-4a49-919b-564b2c019efd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:18.754774 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.754745 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\") pod \"be6a3311-b2a1-4a49-919b-564b2c019efd\" (UID: \"be6a3311-b2a1-4a49-919b-564b2c019efd\") " Oct 30 22:11:18.754996 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.754982 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.755055 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.754999 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d59f\" (UniqueName: \"kubernetes.io/projected/be6a3311-b2a1-4a49-919b-564b2c019efd-kube-api-access-9d59f\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.755055 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.755010 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be6a3311-b2a1-4a49-919b-564b2c019efd-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.755126 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.755083 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca" pod "be6a3311-b2a1-4a49-919b-564b2c019efd" (UID: "be6a3311-b2a1-4a49-919b-564b2c019efd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:18.794850 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.794815 4921 generic.go:356] "Generic (PLEG): container finished" podID="be6a3311-b2a1-4a49-919b-564b2c019efd" containerID="4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94" exitCode=0 Oct 30 22:11:18.794995 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.794874 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" Oct 30 22:11:18.794995 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.794879 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" event={"ID":"be6a3311-b2a1-4a49-919b-564b2c019efd","Type":"ContainerDied","Data":"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94"} Oct 30 22:11:18.795090 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.795014 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9" event={"ID":"be6a3311-b2a1-4a49-919b-564b2c019efd","Type":"ContainerDied","Data":"bf50b2f4bcb02a4b5aa2162e90c34fb6fda23bffba58c5fe9dae2061caa76df9"} Oct 30 22:11:18.795090 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.795050 4921 scope.go:122] "RemoveContainer" containerID="4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94" Oct 30 22:11:18.805490 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.805469 4921 scope.go:122] "RemoveContainer" containerID="4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94" Oct 30 22:11:18.805815 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:11:18.805787 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94\": container with ID starting with 4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94 not found: ID does not exist" containerID="4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94" Oct 30 22:11:18.805901 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.805826 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94"} err="failed to get container status \"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94\": rpc error: code = NotFound desc = could not find container \"4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94\": container with ID starting with 4768447349cf139c9d2945a6026a590970e89a4d6db601e0ef675258020cae94 not found: ID does not exist" Oct 30 22:11:18.855892 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.855867 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be6a3311-b2a1-4a49-919b-564b2c019efd-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.957841 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.957811 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:11:18.961811 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:18.961777 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dc7946-845cf57897-q79z9"] Oct 30 22:11:20.167886 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:20.167855 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6a3311-b2a1-4a49-919b-564b2c019efd" path="/var/lib/kubelet/pods/be6a3311-b2a1-4a49-919b-564b2c019efd/volumes" Oct 30 22:11:25.415373 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.415300 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" podUID="1543c9ee-5fd7-427a-871a-776c58ad5420" containerName="main" containerID="cri-o://2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5" gracePeriod=45 Oct 30 22:11:25.679961 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.679889 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:11:25.728009 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.727969 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728111 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\" (UniqueName: \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728137 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728156 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728181 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\" (UniqueName: \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728209 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\") pod \"1543c9ee-5fd7-427a-871a-776c58ad5420\" (UID: \"1543c9ee-5fd7-427a-871a-776c58ad5420\") " Oct 30 22:11:25.728556 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728511 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:25.728556 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728525 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:25.728556 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728535 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:25.728733 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728704 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.728733 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728720 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.728841 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.728734 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.730538 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.730507 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "kube-api-access-ntp42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:25.730676 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.730560 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:25.730676 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.730567 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs" pod "1543c9ee-5fd7-427a-871a-776c58ad5420" (UID: "1543c9ee-5fd7-427a-871a-776c58ad5420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:25.829953 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.829917 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1543c9ee-5fd7-427a-871a-776c58ad5420-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.829953 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.829950 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntp42\" (UniqueName: \"kubernetes.io/projected/1543c9ee-5fd7-427a-871a-776c58ad5420-kube-api-access-ntp42\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.830160 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.829960 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1543c9ee-5fd7-427a-871a-776c58ad5420-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:25.849243 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.849210 4921 generic.go:356] "Generic (PLEG): container finished" podID="1543c9ee-5fd7-427a-871a-776c58ad5420" containerID="2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5" exitCode=0 Oct 30 22:11:25.849469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.849261 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" Oct 30 22:11:25.849469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.849265 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" event={"ID":"1543c9ee-5fd7-427a-871a-776c58ad5420","Type":"ContainerDied","Data":"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5"} Oct 30 22:11:25.849469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.849378 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc" event={"ID":"1543c9ee-5fd7-427a-871a-776c58ad5420","Type":"ContainerDied","Data":"a4069fd6960ad10163610e63a76d384d0cd5d365ba03f6a0f43ed9fa67f450af"} Oct 30 22:11:25.849469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.849404 4921 scope.go:122] "RemoveContainer" containerID="2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5" Oct 30 22:11:25.860025 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.860005 4921 scope.go:122] "RemoveContainer" containerID="2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5" Oct 30 22:11:25.860358 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:11:25.860332 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5\": container with ID starting with 2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5 not found: ID does not exist" containerID="2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5" Oct 30 22:11:25.860476 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:25.860367 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5"} err="failed to get container status \"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5\": rpc error: code = NotFound desc = could not find container \"2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5\": container with ID starting with 2a6114a89952240bcc08f00b4a24c4b063f7288c70f93a27ce8dc4c767c122e5 not found: ID does not exist" Oct 30 22:11:26.019591 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:26.019552 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:11:26.023655 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:26.023618 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dc7946/simulator-single-main-dc7946-kserve-55f4f9bbbf-lfhpc"] Oct 30 22:11:26.168943 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:26.168907 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1543c9ee-5fd7-427a-871a-776c58ad5420" path="/var/lib/kubelet/pods/1543c9ee-5fd7-427a-871a-776c58ad5420/volumes" Oct 30 22:11:26.983573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:26.983540 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:11:26.996372 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:26.996305 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:11:27.042275 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:27.042246 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:11:27.053420 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:27.053393 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:11:36.316227 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316182 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:11:36.316728 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316713 4921 cpu_manager.go:414] "containerMap: removing container" podUID="1543c9ee-5fd7-427a-871a-776c58ad5420" containerName="main" Oct 30 22:11:36.316780 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316729 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1543c9ee-5fd7-427a-871a-776c58ad5420" containerName="main" Oct 30 22:11:36.316780 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316742 4921 cpu_manager.go:414] "containerMap: removing container" podUID="be6a3311-b2a1-4a49-919b-564b2c019efd" containerName="maas-api" Oct 30 22:11:36.316780 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316747 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6a3311-b2a1-4a49-919b-564b2c019efd" containerName="maas-api" Oct 30 22:11:36.316885 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316834 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1543c9ee-5fd7-427a-871a-776c58ad5420" containerName="main" Oct 30 22:11:36.316885 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.316847 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6a3311-b2a1-4a49-919b-564b2c019efd" containerName="maas-api" Oct 30 22:11:36.320069 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.320050 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.422067 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.422036 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fb2c6f18-kserve-self-signed-certs\"" Oct 30 22:11:36.425178 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.425144 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:11:36.443170 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443141 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.443295 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443185 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.443387 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443363 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.443465 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443454 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.443503 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443473 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7qg\" (UniqueName: \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.443503 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.443491 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544674 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544638 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544674 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544678 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7qg\" (UniqueName: \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544702 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544729 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544763 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.544978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.544852 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.545208 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.545160 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.545456 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.545412 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.545628 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.545519 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.547233 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.547204 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.547699 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.547676 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.602369 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.602277 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7qg\" (UniqueName: \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\") pod \"e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:36.633073 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:36.633031 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:37.024782 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:37.024751 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:11:37.027423 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:11:37.027391 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3fd02db_99af_4255_a90d_3e0e93b6ba4e.slice/crio-4d601c5332055a374a20bb9c4fc800eacfffdfa5d187c307d8cad86722164293 WatchSource:0}: Error finding container 4d601c5332055a374a20bb9c4fc800eacfffdfa5d187c307d8cad86722164293: Status 404 returned error can't find the container with id 4d601c5332055a374a20bb9c4fc800eacfffdfa5d187c307d8cad86722164293 Oct 30 22:11:37.543535 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:11:37.543493 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:11:37.929525 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:37.929428 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" event={"ID":"d3fd02db-99af-4255-a90d-3e0e93b6ba4e","Type":"ContainerStarted","Data":"029d4e20c2a4de3b975ba762f1303b4b450262290eb6ca038b98a89e563006df"} Oct 30 22:11:37.929525 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:37.929474 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" event={"ID":"d3fd02db-99af-4255-a90d-3e0e93b6ba4e","Type":"ContainerStarted","Data":"4d601c5332055a374a20bb9c4fc800eacfffdfa5d187c307d8cad86722164293"} Oct 30 22:11:38.046471 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:38.046410 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" podStartSLOduration=2.046394158 podStartE2EDuration="2.046394158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:38.043739458 +0000 UTC m=+3612.026439873" watchObservedRunningTime="2026-10-30 22:11:38.046394158 +0000 UTC m=+3612.029094576" Oct 30 22:11:38.633185 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:38.633140 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:46.633372 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:46.633333 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:46.645966 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:46.645939 4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:11:47.022428 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:11:47.022394 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:12:07.541672 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:12:07.541589 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:12:37.543794 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:12:37.543754 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:13:07.540808 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:13:07.540764 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:13:15.411940 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:15.411902 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:13:30.436336 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.434544 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" podUID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" containerName="main" containerID="cri-o://029d4e20c2a4de3b975ba762f1303b4b450262290eb6ca038b98a89e563006df" gracePeriod=45 Oct 30 22:13:30.744127 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.743903 4921 generic.go:356] "Generic (PLEG): container finished" podID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" containerID="029d4e20c2a4de3b975ba762f1303b4b450262290eb6ca038b98a89e563006df" exitCode=0 Oct 30 22:13:30.744127 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.744053 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" event={"ID":"d3fd02db-99af-4255-a90d-3e0e93b6ba4e","Type":"ContainerDied","Data":"029d4e20c2a4de3b975ba762f1303b4b450262290eb6ca038b98a89e563006df"} Oct 30 22:13:30.810816 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.810786 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:13:30.885554 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885520 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885731 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885573 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885731 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885605 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885731 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885636 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885731 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885691 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\" (UniqueName: \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885962 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885780 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:30.885962 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885787 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\") pod \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\" (UID: \"d3fd02db-99af-4255-a90d-3e0e93b6ba4e\") " Oct 30 22:13:30.885962 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885923 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:30.886110 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.885966 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:30.886150 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.886124 4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tmp-dir\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.886150 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.886136 4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-home\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.886150 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.886144 4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-model-cache\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.887966 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.887939 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:30.887966 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.887950 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:30.888110 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.887974 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg" pod "d3fd02db-99af-4255-a90d-3e0e93b6ba4e" (UID: "d3fd02db-99af-4255-a90d-3e0e93b6ba4e"). InnerVolumeSpecName "kube-api-access-dt7qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:30.987602 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.987528 4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-dshm\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.987602 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.987554 4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-tls-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:30.987602 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:30.987565 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7qg\" (UniqueName: \"kubernetes.io/projected/d3fd02db-99af-4255-a90d-3e0e93b6ba4e-kube-api-access-dt7qg\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:31.750497 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:31.750465 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" Oct 30 22:13:31.750497 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:31.750481 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh" event={"ID":"d3fd02db-99af-4255-a90d-3e0e93b6ba4e","Type":"ContainerDied","Data":"4d601c5332055a374a20bb9c4fc800eacfffdfa5d187c307d8cad86722164293"} Oct 30 22:13:31.751015 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:31.750539 4921 scope.go:122] "RemoveContainer" containerID="029d4e20c2a4de3b975ba762f1303b4b450262290eb6ca038b98a89e563006df" Oct 30 22:13:31.914088 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:31.914054 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:13:31.963373 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:31.963289 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fb2c6f18-kserve-7fd557f47c-nt5mh"] Oct 30 22:13:32.167347 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:13:32.167253 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" path="/var/lib/kubelet/pods/d3fd02db-99af-4255-a90d-3e0e93b6ba4e/volumes" Oct 30 22:13:37.539324 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:13:37.539225 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:14:07.540112 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:14:07.540075 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:14:09.038268 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.038234 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:09.038740 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.038725 4921 cpu_manager.go:414] "containerMap: removing container" podUID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" containerName="main" Oct 30 22:14:09.038787 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.038741 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" containerName="main" Oct 30 22:14:09.038835 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.038827 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fd02db-99af-4255-a90d-3e0e93b6ba4e" containerName="main" Oct 30 22:14:09.041634 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.041611 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.055456 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.055432 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2"] Oct 30 22:14:09.059032 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.059016 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.096026 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.096005 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.096176 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.096047 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.096176 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.096073 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8s8\" (UniqueName: \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.096302 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.096211 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.102257 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.102216 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:14:09.142637 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.142609 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:14:09.142781 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.142699 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wqlw4\"" Oct 30 22:14:09.145859 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.145835 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:09.148269 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.148250 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2"] Oct 30 22:14:09.197506 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197477 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-maas-api-tls\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.197648 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197616 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.197708 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197658 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.197708 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197682 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q5v\" (UniqueName: \"kubernetes.io/projected/7ed3377a-1239-4898-9900-fe452ec94760-kube-api-access-m8q5v\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.197809 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197790 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8s8\" (UniqueName: \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.197921 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197899 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3377a-1239-4898-9900-fe452ec94760-openshift-service-ca\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.197981 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.197949 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-metrics-certs\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.198051 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.198033 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.198273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.198256 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.200372 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.200352 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.200440 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.200352 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.255889 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.255855 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8s8\" (UniqueName: \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\") pod \"maas-api-5f994fb758-nm9vn\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.299135 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.299056 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-maas-api-tls\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.299277 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.299160 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q5v\" (UniqueName: \"kubernetes.io/projected/7ed3377a-1239-4898-9900-fe452ec94760-kube-api-access-m8q5v\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.299277 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.299179 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3377a-1239-4898-9900-fe452ec94760-openshift-service-ca\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.299277 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.299201 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-metrics-certs\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.299978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.299955 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3377a-1239-4898-9900-fe452ec94760-openshift-service-ca\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.301535 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.301515 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-maas-api-tls\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.301673 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.301654 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed3377a-1239-4898-9900-fe452ec94760-metrics-certs\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.355042 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.355015 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:09.357642 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.357615 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q5v\" (UniqueName: \"kubernetes.io/projected/7ed3377a-1239-4898-9900-fe452ec94760-kube-api-access-m8q5v\") pod \"maas-api-5fb4c4694-zqrl2\" (UID: \"7ed3377a-1239-4898-9900-fe452ec94760\") " pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.372192 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.372166 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:09.789325 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:14:09.789275 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8117bf68_d3c9_4f95_bd04_3576a2584de3.slice/crio-b751d09a8cfb412b0ba9fbbc01c91c06856cccfe575ea695aa4519ccdacdd87f WatchSource:0}: Error finding container b751d09a8cfb412b0ba9fbbc01c91c06856cccfe575ea695aa4519ccdacdd87f: Status 404 returned error can't find the container with id b751d09a8cfb412b0ba9fbbc01c91c06856cccfe575ea695aa4519ccdacdd87f Oct 30 22:14:09.789713 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.789688 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:09.793726 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:09.793704 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2"] Oct 30 22:14:09.825163 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:14:09.825137 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed3377a_1239_4898_9900_fe452ec94760.slice/crio-d48f85ba5c7c9e733ef2c9aa30310d1b5436782002d13b9b4fefea5fd5cae9d3 WatchSource:0}: Error finding container d48f85ba5c7c9e733ef2c9aa30310d1b5436782002d13b9b4fefea5fd5cae9d3: Status 404 returned error can't find the container with id d48f85ba5c7c9e733ef2c9aa30310d1b5436782002d13b9b4fefea5fd5cae9d3 Oct 30 22:14:10.011246 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:10.011212 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" event={"ID":"7ed3377a-1239-4898-9900-fe452ec94760","Type":"ContainerStarted","Data":"d48f85ba5c7c9e733ef2c9aa30310d1b5436782002d13b9b4fefea5fd5cae9d3"} Oct 30 22:14:10.012523 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:10.012493 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" event={"ID":"8117bf68-d3c9-4f95-bd04-3576a2584de3","Type":"ContainerStarted","Data":"b751d09a8cfb412b0ba9fbbc01c91c06856cccfe575ea695aa4519ccdacdd87f"} Oct 30 22:14:11.020265 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.020226 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" event={"ID":"7ed3377a-1239-4898-9900-fe452ec94760","Type":"ContainerStarted","Data":"21530d564e0730bf9ae58c7ac96abdc821c6287ef1957e179f0fed6c4ad9d50f"} Oct 30 22:14:11.020729 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.020291 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:11.022075 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.022049 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" event={"ID":"8117bf68-d3c9-4f95-bd04-3576a2584de3","Type":"ContainerStarted","Data":"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d"} Oct 30 22:14:11.022198 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.022118 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:11.139724 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.139672 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" podStartSLOduration=1.2780102420000001 podStartE2EDuration="2.139658263s" totalImagesPullingTime="861.648021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:11.136428324 +0000 UTC m=+3765.119128740" watchObservedRunningTime="2026-10-30 22:14:11.139658263 +0000 UTC m=+3765.122358674" Oct 30 22:14:11.251595 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:11.251534 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" podStartSLOduration=2.349392504 podStartE2EDuration="3.251508898s" totalImagesPullingTime="902.116394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:11.25063404 +0000 UTC m=+3765.233334458" watchObservedRunningTime="2026-10-30 22:14:11.251508898 +0000 UTC m=+3765.234209317" Oct 30 22:14:17.033661 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.033633 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fb4c4694-zqrl2" Oct 30 22:14:17.034126 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.033996 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:17.221340 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.221275 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:17.221521 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.221496 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" podUID="8117bf68-d3c9-4f95-bd04-3576a2584de3" containerName="maas-api" containerID="cri-o://8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d" gracePeriod=30 Oct 30 22:14:17.495761 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.495731 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:17.583341 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.583300 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\" (UniqueName: \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\") pod \"8117bf68-d3c9-4f95-bd04-3576a2584de3\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " Oct 30 22:14:17.583530 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.583370 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\") pod \"8117bf68-d3c9-4f95-bd04-3576a2584de3\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " Oct 30 22:14:17.583530 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.583422 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\") pod \"8117bf68-d3c9-4f95-bd04-3576a2584de3\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " Oct 30 22:14:17.583530 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.583497 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\") pod \"8117bf68-d3c9-4f95-bd04-3576a2584de3\" (UID: \"8117bf68-d3c9-4f95-bd04-3576a2584de3\") " Oct 30 22:14:17.583864 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.583840 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca" pod "8117bf68-d3c9-4f95-bd04-3576a2584de3" (UID: "8117bf68-d3c9-4f95-bd04-3576a2584de3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:17.585469 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.585438 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs" pod "8117bf68-d3c9-4f95-bd04-3576a2584de3" (UID: "8117bf68-d3c9-4f95-bd04-3576a2584de3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:17.585548 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.585476 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls" pod "8117bf68-d3c9-4f95-bd04-3576a2584de3" (UID: "8117bf68-d3c9-4f95-bd04-3576a2584de3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:17.585548 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.585515 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8" pod "8117bf68-d3c9-4f95-bd04-3576a2584de3" (UID: "8117bf68-d3c9-4f95-bd04-3576a2584de3"). InnerVolumeSpecName "kube-api-access-jm8s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:17.684395 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.684364 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8117bf68-d3c9-4f95-bd04-3576a2584de3-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:17.684395 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.684390 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:17.684395 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.684399 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8s8\" (UniqueName: \"kubernetes.io/projected/8117bf68-d3c9-4f95-bd04-3576a2584de3-kube-api-access-jm8s8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:17.684628 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:17.684407 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8117bf68-d3c9-4f95-bd04-3576a2584de3-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:18.077196 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.077162 4921 generic.go:356] "Generic (PLEG): container finished" podID="8117bf68-d3c9-4f95-bd04-3576a2584de3" containerID="8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d" exitCode=0 Oct 30 22:14:18.077700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.077228 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" Oct 30 22:14:18.077700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.077256 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" event={"ID":"8117bf68-d3c9-4f95-bd04-3576a2584de3","Type":"ContainerDied","Data":"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d"} Oct 30 22:14:18.077700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.077326 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn" event={"ID":"8117bf68-d3c9-4f95-bd04-3576a2584de3","Type":"ContainerDied","Data":"b751d09a8cfb412b0ba9fbbc01c91c06856cccfe575ea695aa4519ccdacdd87f"} Oct 30 22:14:18.077700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.077370 4921 scope.go:122] "RemoveContainer" containerID="8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d" Oct 30 22:14:18.087509 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.087379 4921 scope.go:122] "RemoveContainer" containerID="8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d" Oct 30 22:14:18.087759 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:14:18.087697 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d\": container with ID starting with 8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d not found: ID does not exist" containerID="8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d" Oct 30 22:14:18.087759 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.087733 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d"} err="failed to get container status \"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d\": rpc error: code = NotFound desc = could not find container \"8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d\": container with ID starting with 8b5d6bd559e367b39242677d458284bbc0322da592287279d707811892e89f9d not found: ID does not exist" Oct 30 22:14:18.239965 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.239931 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:18.289853 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:18.289816 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f994fb758-nm9vn"] Oct 30 22:14:20.167766 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:20.167730 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8117bf68-d3c9-4f95-bd04-3576a2584de3" path="/var/lib/kubelet/pods/8117bf68-d3c9-4f95-bd04-3576a2584de3/volumes" Oct 30 22:14:24.272115 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.272083 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:14:24.272588 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.272557 4921 cpu_manager.go:414] "containerMap: removing container" podUID="8117bf68-d3c9-4f95-bd04-3576a2584de3" containerName="maas-api" Oct 30 22:14:24.272588 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.272572 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8117bf68-d3c9-4f95-bd04-3576a2584de3" containerName="maas-api" Oct 30 22:14:24.272692 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.272681 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8117bf68-d3c9-4f95-bd04-3576a2584de3" containerName="maas-api" Oct 30 22:14:24.277136 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.277117 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.330511 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.330473 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-87bd1eb1-serving-cert\"" Oct 30 22:14:24.382694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.382662 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:14:24.450131 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.450100 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.450293 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.450148 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxdx\" (UniqueName: \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.450368 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.450288 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.450368 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.450335 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.550963 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.550872 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.550963 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.550952 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.551212 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.550981 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxdx\" (UniqueName: \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.551212 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:14:24.551024 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-serving-cert: secret "maas-api-e2e-ait-87bd1eb1-serving-cert" not found Oct 30 22:14:24.551212 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.551046 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.551212 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:14:24.551106 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls podName:483bfeab-803f-4f41-b35c-9bd5a839003a nodeName:}" failed. No retries permitted until 2026-10-30 22:14:25.05108304 +0000 UTC m=+3779.033783450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls") pod "maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" (UID: "483bfeab-803f-4f41-b35c-9bd5a839003a") : secret "maas-api-e2e-ait-87bd1eb1-serving-cert" not found Oct 30 22:14:24.551769 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.551746 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.553618 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.553592 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:24.607976 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:24.607950 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxdx\" (UniqueName: \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:25.055122 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:25.055086 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:25.057531 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:25.057508 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") pod \"maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:25.190097 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:25.190069 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:25.577293 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:25.577249 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:14:25.583391 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:14:25.583365 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483bfeab_803f_4f41_b35c_9bd5a839003a.slice/crio-3d846aee51685a9166cb6dcb978897e4c2ea5f2444e8f4f06a7d2e015b7e0a05 WatchSource:0}: Error finding container 3d846aee51685a9166cb6dcb978897e4c2ea5f2444e8f4f06a7d2e015b7e0a05: Status 404 returned error can't find the container with id 3d846aee51685a9166cb6dcb978897e4c2ea5f2444e8f4f06a7d2e015b7e0a05 Oct 30 22:14:26.133473 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:26.133442 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" event={"ID":"483bfeab-803f-4f41-b35c-9bd5a839003a","Type":"ContainerStarted","Data":"3d846aee51685a9166cb6dcb978897e4c2ea5f2444e8f4f06a7d2e015b7e0a05"} Oct 30 22:14:27.141068 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:27.141028 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" event={"ID":"483bfeab-803f-4f41-b35c-9bd5a839003a","Type":"ContainerStarted","Data":"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b"} Oct 30 22:14:27.141561 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:27.141251 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:27.259074 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:27.259021 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" podStartSLOduration=2.427812063 podStartE2EDuration="3.259002238s" totalImagesPullingTime="831.190175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.25578457 +0000 UTC m=+3781.238484991" watchObservedRunningTime="2026-10-30 22:14:27.259002238 +0000 UTC m=+3781.241702654" Oct 30 22:14:33.152568 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:14:33.152536 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:14:37.539791 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:14:37.539754 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:15:03.775392 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:03.775361 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:15:03.775893 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:03.775583 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" podUID="483bfeab-803f-4f41-b35c-9bd5a839003a" containerName="maas-api" containerID="cri-o://b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b" gracePeriod=30 Oct 30 22:15:04.038949 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.038887 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:15:04.221455 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221419 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\") pod \"483bfeab-803f-4f41-b35c-9bd5a839003a\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " Oct 30 22:15:04.221639 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221465 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\") pod \"483bfeab-803f-4f41-b35c-9bd5a839003a\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " Oct 30 22:15:04.221639 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221580 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\" (UniqueName: \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\") pod \"483bfeab-803f-4f41-b35c-9bd5a839003a\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " Oct 30 22:15:04.221768 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221639 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") pod \"483bfeab-803f-4f41-b35c-9bd5a839003a\" (UID: \"483bfeab-803f-4f41-b35c-9bd5a839003a\") " Oct 30 22:15:04.221861 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221838 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca" pod "483bfeab-803f-4f41-b35c-9bd5a839003a" (UID: "483bfeab-803f-4f41-b35c-9bd5a839003a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:04.222020 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.221994 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483bfeab-803f-4f41-b35c-9bd5a839003a-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.223596 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.223565 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs" pod "483bfeab-803f-4f41-b35c-9bd5a839003a" (UID: "483bfeab-803f-4f41-b35c-9bd5a839003a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:04.223694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.223615 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls" pod "483bfeab-803f-4f41-b35c-9bd5a839003a" (UID: "483bfeab-803f-4f41-b35c-9bd5a839003a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:04.223694 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.223679 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx" pod "483bfeab-803f-4f41-b35c-9bd5a839003a" (UID: "483bfeab-803f-4f41-b35c-9bd5a839003a"). InnerVolumeSpecName "kube-api-access-htxdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:04.323383 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.323294 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.323383 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.323338 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/483bfeab-803f-4f41-b35c-9bd5a839003a-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.323383 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.323349 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxdx\" (UniqueName: \"kubernetes.io/projected/483bfeab-803f-4f41-b35c-9bd5a839003a-kube-api-access-htxdx\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:04.346875 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.346836 4921 generic.go:356] "Generic (PLEG): container finished" podID="483bfeab-803f-4f41-b35c-9bd5a839003a" containerID="b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b" exitCode=0 Oct 30 22:15:04.347050 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.346885 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" Oct 30 22:15:04.347050 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.346918 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" event={"ID":"483bfeab-803f-4f41-b35c-9bd5a839003a","Type":"ContainerDied","Data":"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b"} Oct 30 22:15:04.347050 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.346986 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v" event={"ID":"483bfeab-803f-4f41-b35c-9bd5a839003a","Type":"ContainerDied","Data":"3d846aee51685a9166cb6dcb978897e4c2ea5f2444e8f4f06a7d2e015b7e0a05"} Oct 30 22:15:04.347050 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.347012 4921 scope.go:122] "RemoveContainer" containerID="b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b" Oct 30 22:15:04.357584 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.357565 4921 scope.go:122] "RemoveContainer" containerID="b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b" Oct 30 22:15:04.357849 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:15:04.357827 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b\": container with ID starting with b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b not found: ID does not exist" containerID="b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b" Oct 30 22:15:04.357932 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.357862 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b"} err="failed to get container status \"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b\": rpc error: code = NotFound desc = could not find container \"b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b\": container with ID starting with b07c55180a0a7065ca929d895fe07f066c39d65e095a2c95f639603d8be9c10b not found: ID does not exist" Oct 30 22:15:04.510185 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.510146 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:15:04.514124 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:04.514094 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87bd1eb1-584b67958f-jzr6v"] Oct 30 22:15:06.167095 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:06.167065 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="483bfeab-803f-4f41-b35c-9bd5a839003a" path="/var/lib/kubelet/pods/483bfeab-803f-4f41-b35c-9bd5a839003a/volumes" Oct 30 22:15:07.539153 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:15:07.539070 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:15:23.530635 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.530603 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:15:23.531049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.531037 4921 cpu_manager.go:414] "containerMap: removing container" podUID="483bfeab-803f-4f41-b35c-9bd5a839003a" containerName="maas-api" Oct 30 22:15:23.531102 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.531049 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="483bfeab-803f-4f41-b35c-9bd5a839003a" containerName="maas-api" Oct 30 22:15:23.531139 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.531125 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="483bfeab-803f-4f41-b35c-9bd5a839003a" containerName="maas-api" Oct 30 22:15:23.535724 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.535706 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.588906 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.588882 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-6e7187-serving-cert\"" Oct 30 22:15:23.599373 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.599342 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.599494 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.599403 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.599561 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.599491 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.599680 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.599564 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hm8\" (UniqueName: \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.641196 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.641168 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:15:23.700449 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.700423 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.700599 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.700488 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hm8\" (UniqueName: \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.700599 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.700555 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.700599 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:15:23.700571 4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-serving-cert: secret "maas-api-e2e-authcfg-http-6e7187-serving-cert" not found Oct 30 22:15:23.700599 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.700585 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.700837 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:15:23.700655 4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls podName:86f146c5-dfe4-4443-867b-62a560ec183b nodeName:}" failed. No retries permitted until 2026-10-30 22:15:24.200629719 +0000 UTC m=+3838.183330127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls") pod "maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" (UID: "86f146c5-dfe4-4443-867b-62a560ec183b") : secret "maas-api-e2e-authcfg-http-6e7187-serving-cert" not found Oct 30 22:15:23.701257 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.701234 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.702989 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.702970 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:23.757647 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:23.757622 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hm8\" (UniqueName: \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:24.205593 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:24.205558 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:24.207783 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:24.207759 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:24.448519 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:24.448484 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:24.831765 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:24.831736 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:15:24.833641 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:15:24.833605 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f146c5_dfe4_4443_867b_62a560ec183b.slice/crio-2f98e954a6a491e3e8365c3f565aa67fceba01bfaf8f2e28a15f9f3d9a7a8eaf WatchSource:0}: Error finding container 2f98e954a6a491e3e8365c3f565aa67fceba01bfaf8f2e28a15f9f3d9a7a8eaf: Status 404 returned error can't find the container with id 2f98e954a6a491e3e8365c3f565aa67fceba01bfaf8f2e28a15f9f3d9a7a8eaf Oct 30 22:15:24.835344 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:24.835327 4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:15:25.474984 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:25.474949 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" event={"ID":"86f146c5-dfe4-4443-867b-62a560ec183b","Type":"ContainerStarted","Data":"2f98e954a6a491e3e8365c3f565aa67fceba01bfaf8f2e28a15f9f3d9a7a8eaf"} Oct 30 22:15:26.482327 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:26.482285 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" event={"ID":"86f146c5-dfe4-4443-867b-62a560ec183b","Type":"ContainerStarted","Data":"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1"} Oct 30 22:15:26.482803 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:26.482398 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:26.616049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:26.615988 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" podStartSLOduration=2.805006635 podStartE2EDuration="3.615974488s" totalImagesPullingTime="810.967853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:26.613892665 +0000 UTC m=+3840.596593083" watchObservedRunningTime="2026-10-30 22:15:26.615974488 +0000 UTC m=+3840.598674905" Oct 30 22:15:32.493513 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:15:32.493482 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:15:37.540890 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:15:37.540848 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:16:07.539438 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:16:07.539402 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:16:27.030371 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:16:27.030342 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:16:27.041732 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:16:27.041707 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:16:27.087567 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:16:27.087535 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:16:27.098797 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:16:27.098768 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:16:37.541078 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:16:37.540993 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:17:07.540687 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:17:07.540651 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:17:26.217655 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.217626 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:17:26.218165 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.217798 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" podUID="86f146c5-dfe4-4443-867b-62a560ec183b" containerName="maas-api" containerID="cri-o://ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1" gracePeriod=30 Oct 30 22:17:26.476787 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.476717 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:17:26.635562 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.635533 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\") pod \"86f146c5-dfe4-4443-867b-62a560ec183b\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " Oct 30 22:17:26.635740 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.635587 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") pod \"86f146c5-dfe4-4443-867b-62a560ec183b\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " Oct 30 22:17:26.635740 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.635694 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\" (UniqueName: \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\") pod \"86f146c5-dfe4-4443-867b-62a560ec183b\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " Oct 30 22:17:26.635831 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.635746 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\") pod \"86f146c5-dfe4-4443-867b-62a560ec183b\" (UID: \"86f146c5-dfe4-4443-867b-62a560ec183b\") " Oct 30 22:17:26.635928 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.635904 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca" pod "86f146c5-dfe4-4443-867b-62a560ec183b" (UID: "86f146c5-dfe4-4443-867b-62a560ec183b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:26.636136 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.636105 4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f146c5-dfe4-4443-867b-62a560ec183b-openshift-service-ca\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.637747 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.637716 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls" pod "86f146c5-dfe4-4443-867b-62a560ec183b" (UID: "86f146c5-dfe4-4443-867b-62a560ec183b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:26.637876 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.637769 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8" pod "86f146c5-dfe4-4443-867b-62a560ec183b" (UID: "86f146c5-dfe4-4443-867b-62a560ec183b"). InnerVolumeSpecName "kube-api-access-v6hm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:26.637876 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.637816 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs" pod "86f146c5-dfe4-4443-867b-62a560ec183b" (UID: "86f146c5-dfe4-4443-867b-62a560ec183b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:26.737401 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.737240 4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-maas-api-tls\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.737401 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.737352 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6hm8\" (UniqueName: \"kubernetes.io/projected/86f146c5-dfe4-4443-867b-62a560ec183b-kube-api-access-v6hm8\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:26.737401 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:26.737374 4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86f146c5-dfe4-4443-867b-62a560ec183b-metrics-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:27.200015 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.199978 4921 generic.go:356] "Generic (PLEG): container finished" podID="86f146c5-dfe4-4443-867b-62a560ec183b" containerID="ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1" exitCode=0 Oct 30 22:17:27.200194 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.200030 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" Oct 30 22:17:27.200194 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.200073 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" event={"ID":"86f146c5-dfe4-4443-867b-62a560ec183b","Type":"ContainerDied","Data":"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1"} Oct 30 22:17:27.200194 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.200144 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj" event={"ID":"86f146c5-dfe4-4443-867b-62a560ec183b","Type":"ContainerDied","Data":"2f98e954a6a491e3e8365c3f565aa67fceba01bfaf8f2e28a15f9f3d9a7a8eaf"} Oct 30 22:17:27.200194 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.200177 4921 scope.go:122] "RemoveContainer" containerID="ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1" Oct 30 22:17:27.210495 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.210479 4921 scope.go:122] "RemoveContainer" containerID="ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1" Oct 30 22:17:27.210811 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:17:27.210792 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1\": container with ID starting with ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1 not found: ID does not exist" containerID="ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1" Oct 30 22:17:27.210890 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.210819 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1"} err="failed to get container status \"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1\": rpc error: code = NotFound desc = could not find container \"ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1\": container with ID starting with ba09761f3c73ddebba303bb0f68d6b4b180ab5607141dae18e1d8d278ed6dcf1 not found: ID does not exist" Oct 30 22:17:27.363152 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.363123 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:17:27.412512 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:27.412488 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-6e7187-6746ddfc8d-p4bjj"] Oct 30 22:17:28.168164 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:17:28.168087 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f146c5-dfe4-4443-867b-62a560ec183b" path="/var/lib/kubelet/pods/86f146c5-dfe4-4443-867b-62a560ec183b/volumes" Oct 30 22:17:37.540466 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:17:37.540421 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:18:07.539681 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:18:07.539604 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:18:29.720340 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:18:29.719418 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-qm6kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:18:29.720340 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:18:29.719496 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:18:37.539378 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:18:37.539344 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:18:54.720297 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:18:54.720262 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848f4b5756-qpp8k_e570d1ef-3aea-4199-8784-919aac381ad6/authorino/0.log" Oct 30 22:19:07.539389 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:19:07.539347 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:19:36.054493 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:36.054463 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-nl2n2_dcf552bb-d943-4039-9f25-fd35df68f826/manager/0.log" Oct 30 22:19:36.339588 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:36.339502 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fw5z9_4ba4bda3-21eb-4b27-9c77-2cc7c05707c9/manager/0.log" Oct 30 22:19:37.127983 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:37.127918 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:19:37.539529 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:19:37.539494 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:19:37.580632 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:37.580600 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vlp2m_74f023e8-abb9-482b-9597-f4e9e60831d8/server/0.log" Oct 30 22:19:39.492754 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:39.492723 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fb4c4694-zqrl2_7ed3377a-1239-4898-9900-fe452ec94760/maas-api/0.log" Oct 30 22:19:40.453164 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:40.453137 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-zbgnw_cef2e5c6-83a4-4880-b42d-90630d6bd804/postgres/0.log" Oct 30 22:19:52.016875 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:52.016842 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848f4b5756-qpp8k_e570d1ef-3aea-4199-8784-919aac381ad6/authorino/0.log" Oct 30 22:19:52.484257 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:52.484205 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vj998_bc5307ad-9cdc-4c05-90e0-e7c43c1a628e/manager/0.log" Oct 30 22:19:52.744203 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:52.744103 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-shhw8_7df10ec4-8bfc-44b6-98b1-5109f7380c48/manager/0.log" Oct 30 22:19:53.262181 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:53.262151 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-czknh_6fa66274-3adf-4103-a622-04048b8732d3/manager/0.log" Oct 30 22:19:53.850788 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:53.850754 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xq4qm_1bef682f-cb5d-48c3-a615-7fb2418d1ca8/manager/0.log" Oct 30 22:19:54.659765 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:19:54.659719 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqblwr_35e7e234-f650-479c-83b9-26e9e9195867/istio-proxy/0.log" Oct 30 22:20:05.739608 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:05.739567 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bfzs_d6df4698-b79b-4ef9-9f0f-4a59754b091d/istio-proxy/0.log" Oct 30 22:20:07.067406 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:07.067374 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jq6qc_f76cebce-aa54-4518-ab9d-5e5530a1c3c2/istio-proxy/0.log" Oct 30 22:20:07.394523 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:07.394446 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-qm6kg_f5bcd4b4-0868-47eb-bcc7-3fe245974842/istio-proxy/0.log" Oct 30 22:20:07.538078 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:20:07.538044 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:20:08.520859 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:08.520831 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd_3aeeb498-3174-46e9-a86c-ade577195b67/istio-proxy/0.log" Oct 30 22:20:09.272436 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:09.272410 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dc5bbb4f7-hmkgf_bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/router/0.log" Oct 30 22:20:10.217772 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:10.217740 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q_659904f4-f858-4f39-8269-dd124fb116b7/main/0.log" Oct 30 22:20:10.735957 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:10.735931 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-l7xtj_1a3e99ee-1a34-4531-99e4-5a7a083bce3b/main/0.log" Oct 30 22:20:11.023564 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:11.023485 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dg85r_258abee6-16f7-47a1-8ba8-0cc1b54c00ec/main/0.log" Oct 30 22:20:11.593936 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:11.593902 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rv945_4e68b799-38c3-4d0d-837a-4587ac587861/main/0.log" Oct 30 22:20:11.890946 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:11.890880 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs_2eb29d8b-7e33-4068-9fd5-8ceb7cde6606/main/0.log" Oct 30 22:20:13.201990 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:13.201958 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848f4b5756-qpp8k_e570d1ef-3aea-4199-8784-919aac381ad6/authorino/0.log" Oct 30 22:20:32.643246 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:32.643209 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-nl2n2_dcf552bb-d943-4039-9f25-fd35df68f826/manager/0.log" Oct 30 22:20:32.969934 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:32.969903 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fw5z9_4ba4bda3-21eb-4b27-9c77-2cc7c05707c9/manager/0.log" Oct 30 22:20:33.718957 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:33.718928 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:20:34.089033 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:34.088955 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vlp2m_74f023e8-abb9-482b-9597-f4e9e60831d8/server/0.log" Oct 30 22:20:35.471830 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:35.471802 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fb4c4694-zqrl2_7ed3377a-1239-4898-9900-fe452ec94760/maas-api/0.log" Oct 30 22:20:36.473230 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:36.473203 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-zbgnw_cef2e5c6-83a4-4880-b42d-90630d6bd804/postgres/0.log" Oct 30 22:20:37.538976 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:20:37.538943 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:20:37.700686 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:37.700655 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848f4b5756-qpp8k_e570d1ef-3aea-4199-8784-919aac381ad6/authorino/0.log" Oct 30 22:20:38.179779 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:38.179752 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vj998_bc5307ad-9cdc-4c05-90e0-e7c43c1a628e/manager/0.log" Oct 30 22:20:38.430727 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:38.430633 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-shhw8_7df10ec4-8bfc-44b6-98b1-5109f7380c48/manager/0.log" Oct 30 22:20:38.938096 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:38.938070 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-czknh_6fa66274-3adf-4103-a622-04048b8732d3/manager/0.log" Oct 30 22:20:39.550689 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:39.550654 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xq4qm_1bef682f-cb5d-48c3-a615-7fb2418d1ca8/manager/0.log" Oct 30 22:20:40.184672 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:40.184647 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqblwr_35e7e234-f650-479c-83b9-26e9e9195867/istio-proxy/0.log" Oct 30 22:20:41.211147 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:41.211112 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bfzs_d6df4698-b79b-4ef9-9f0f-4a59754b091d/istio-proxy/0.log" Oct 30 22:20:42.399482 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:42.399450 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jq6qc_f76cebce-aa54-4518-ab9d-5e5530a1c3c2/istio-proxy/0.log" Oct 30 22:20:42.726309 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:42.726277 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-qm6kg_f5bcd4b4-0868-47eb-bcc7-3fe245974842/istio-proxy/0.log" Oct 30 22:20:43.758443 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:43.758410 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd_3aeeb498-3174-46e9-a86c-ade577195b67/istio-proxy/0.log" Oct 30 22:20:44.491969 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:44.491942 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dc5bbb4f7-hmkgf_bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/router/0.log" Oct 30 22:20:45.130667 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:45.130633 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ng29q_659904f4-f858-4f39-8269-dd124fb116b7/main/0.log" Oct 30 22:20:45.595791 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:45.595762 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-l7xtj_1a3e99ee-1a34-4531-99e4-5a7a083bce3b/main/0.log" Oct 30 22:20:45.820282 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:45.820257 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dg85r_258abee6-16f7-47a1-8ba8-0cc1b54c00ec/main/0.log" Oct 30 22:20:46.325100 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:46.325071 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rv945_4e68b799-38c3-4d0d-837a-4587ac587861/main/0.log" Oct 30 22:20:46.600504 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:20:46.600429 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gzqzs_2eb29d8b-7e33-4068-9fd5-8ceb7cde6606/main/0.log" Oct 30 22:21:07.539792 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:21:07.539717 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:21:19.502636 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:19.502607 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:21:20.085883 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:20.085849 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:21:27.072903 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:27.072877 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:21:27.084548 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:27.084524 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:21:27.138599 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:27.138574 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awvm8k_39473baf-db93-41ba-a1cc-734a89501156/extract/0.log" Oct 30 22:21:27.148940 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:21:27.148920 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:21:37.539703 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:21:37.539665 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:22:07.539410 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:22:07.539372 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:22:37.540161 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:22:37.540085 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:22:56.548187 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:22:56.548152 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:22:57.128531 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:22:57.128504 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:23:07.540543 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:23:07.540502 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:23:08.110752 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:08.110716 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mp2j2_503a46bf-3696-4a30-953b-19ed465b5919/global-pull-secret-syncer/0.log" Oct 30 22:23:09.303057 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:09.303029 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-44tx6_e9686531-8650-4b17-a9f8-002a96ba1fb4/connection-checker/0.log" Oct 30 22:23:09.721125 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:09.721097 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4vfv6_efcd9b7e-a9d6-49c1-8792-69fec47961ee/connection-checker/0.log" Oct 30 22:23:10.138847 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:10.138769 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xmkfl_0620bbf3-fe2f-4967-b303-810f585e75e5/connection-checker/0.log" Oct 30 22:23:11.166072 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:11.166035 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z2rbr_16ebb912-daf3-4d1c-b78c-46a14632cca5/konnectivity-agent/0.log" Oct 30 22:23:12.402978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:12.402942 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-220.us-west-1.compute.internal_48089fcbd21ef30454f56e9b574f5b67/haproxy/0.log" Oct 30 22:23:27.343842 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:27.343801 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-848f4b5756-qpp8k_e570d1ef-3aea-4199-8784-919aac381ad6/authorino/0.log" Oct 30 22:23:27.734190 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:27.734160 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vj998_bc5307ad-9cdc-4c05-90e0-e7c43c1a628e/manager/0.log" Oct 30 22:23:28.155553 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:28.155472 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-shhw8_7df10ec4-8bfc-44b6-98b1-5109f7380c48/manager/0.log" Oct 30 22:23:29.027592 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:29.027562 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-czknh_6fa66274-3adf-4103-a622-04048b8732d3/manager/0.log" Oct 30 22:23:29.896171 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:29.896139 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xq4qm_1bef682f-cb5d-48c3-a615-7fb2418d1ca8/manager/0.log" Oct 30 22:23:35.357678 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:35.357641 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-sjz89_7c440558-5078-4b4d-b28a-2c36109ab98e/cluster-monitoring-operator/0.log" Oct 30 22:23:35.503764 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:35.503729 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqblwr_35e7e234-f650-479c-83b9-26e9e9195867/istio-proxy/0.log" Oct 30 22:23:37.287392 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.287342 4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g"] Oct 30 22:23:37.287887 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.287842 4921 cpu_manager.go:414] "containerMap: removing container" podUID="86f146c5-dfe4-4443-867b-62a560ec183b" containerName="maas-api" Oct 30 22:23:37.287887 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.287856 4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f146c5-dfe4-4443-867b-62a560ec183b" containerName="maas-api" Oct 30 22:23:37.287978 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.287943 4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f146c5-dfe4-4443-867b-62a560ec183b" containerName="maas-api" Oct 30 22:23:37.291358 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.291304 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.343086 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.343044 4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfnpp\"/\"default-dockercfg-k28w6\"" Oct 30 22:23:37.378631 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.378593 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-podres\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.378631 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.378633 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-proc\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.378924 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.378764 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-sys\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.378924 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.378843 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvch\" (UniqueName: \"kubernetes.io/projected/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-kube-api-access-scvch\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.379009 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.378934 4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-lib-modules\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.392419 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.392382 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfnpp\"/\"kube-root-ca.crt\"" Oct 30 22:23:37.392419 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.392405 4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfnpp\"/\"openshift-service-ca.crt\"" Oct 30 22:23:37.396013 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.395989 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g"] Oct 30 22:23:37.480000 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.479929 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-lib-modules\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480062 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-podres\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480085 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-proc\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480084 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-lib-modules\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480147 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-proc\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480179 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-sys\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480193 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-podres\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480214 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480221 4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvch\" (UniqueName: \"kubernetes.io/projected/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-kube-api-access-scvch\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.480748 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.480266 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-sys\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.543513 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:23:37.543397 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:23:37.544652 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.544629 4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvch\" (UniqueName: \"kubernetes.io/projected/6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57-kube-api-access-scvch\") pod \"perf-node-gather-daemonset-ngz7g\" (UID: \"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57\") " pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.604512 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.604477 4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:37.983726 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:37.983696 4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g"] Oct 30 22:23:37.985032 ip-10-0-137-220 kubenswrapper[4921]: W1030 22:23:37.985001 4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6cd8d92d_2fa6_44c2_9aec_93fa43ccbb57.slice/crio-32ce65404e7c82cdd877324cc002de5cdf0bc7315ca1540b39b0a49982cddf43 WatchSource:0}: Error finding container 32ce65404e7c82cdd877324cc002de5cdf0bc7315ca1540b39b0a49982cddf43: Status 404 returned error can't find the container with id 32ce65404e7c82cdd877324cc002de5cdf0bc7315ca1540b39b0a49982cddf43 Oct 30 22:23:38.461326 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:38.461269 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" event={"ID":"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57","Type":"ContainerStarted","Data":"75bd7d4a0b7446f05346ad9c936b01174e4791263b6fde3309db3ab420e60e70"} Oct 30 22:23:38.461814 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:38.461358 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" event={"ID":"6cd8d92d-2fa6-44c2-9aec-93fa43ccbb57","Type":"ContainerStarted","Data":"32ce65404e7c82cdd877324cc002de5cdf0bc7315ca1540b39b0a49982cddf43"} Oct 30 22:23:38.461814 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:38.461471 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:38.575305 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:38.575248 4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" podStartSLOduration=1.575231866 podStartE2EDuration="1.575231866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:38.57269531 +0000 UTC m=+4332.555395730" watchObservedRunningTime="2026-10-30 22:23:38.575231866 +0000 UTC m=+4332.557932284" Oct 30 22:23:38.829429 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:38.829341 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bfzs_d6df4698-b79b-4ef9-9f0f-4a59754b091d/istio-proxy/0.log" Oct 30 22:23:39.893549 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:39.893519 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jq6qc_f76cebce-aa54-4518-ab9d-5e5530a1c3c2/istio-proxy/0.log" Oct 30 22:23:40.107251 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:40.107219 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-qm6kg_f5bcd4b4-0868-47eb-bcc7-3fe245974842/istio-proxy/0.log" Oct 30 22:23:40.378132 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:40.378097 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7d46_33465666-e6b5-42fe-838a-e8ea65b65870/node-exporter/0.log" Oct 30 22:23:40.802383 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:40.802349 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7d46_33465666-e6b5-42fe-838a-e8ea65b65870/kube-rbac-proxy/0.log" Oct 30 22:23:40.986933 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:40.986904 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd_3aeeb498-3174-46e9-a86c-ade577195b67/istio-proxy/0.log" Oct 30 22:23:41.264829 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:41.264796 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x7d46_33465666-e6b5-42fe-838a-e8ea65b65870/init-textfile/0.log" Oct 30 22:23:41.669690 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:41.669607 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dc5bbb4f7-hmkgf_bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/router/0.log" Oct 30 22:23:44.478138 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:44.478103 4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfnpp/perf-node-gather-daemonset-ngz7g" Oct 30 22:23:46.028533 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:46.028501 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqblwr_35e7e234-f650-479c-83b9-26e9e9195867/istio-proxy/0.log" Oct 30 22:23:49.223617 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:49.223583 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bfzs_d6df4698-b79b-4ef9-9f0f-4a59754b091d/istio-proxy/0.log" Oct 30 22:23:50.292771 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:50.292740 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jq6qc_f76cebce-aa54-4518-ab9d-5e5530a1c3c2/istio-proxy/0.log" Oct 30 22:23:50.510248 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:50.510213 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-qm6kg_f5bcd4b4-0868-47eb-bcc7-3fe245974842/istio-proxy/0.log" Oct 30 22:23:51.362086 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:51.362051 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd_3aeeb498-3174-46e9-a86c-ade577195b67/istio-proxy/0.log" Oct 30 22:23:51.998853 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:51.998821 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dc5bbb4f7-hmkgf_bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/router/0.log" Oct 30 22:23:53.607041 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:53.607013 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-6bt4k_d478b307-b6de-4b2a-b6cf-3a3b7bbbb414/networking-console-plugin/0.log" Oct 30 22:23:56.358820 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:56.358786 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/3.log" Oct 30 22:23:56.365686 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:23:56.365654 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rvc5s_4b295f49-70ab-4d12-b08c-57f3dfb5324b/console-operator/4.log" Oct 30 22:24:01.851142 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:01.851105 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-6s6mw_d120a1ef-dae4-4199-8149-0ae9660d8dfe/volume-data-source-validator/0.log" Oct 30 22:24:07.539993 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:24:07.539913 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:24:08.056231 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:08.056197 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dspqr_ddc7b666-f4e0-4a1d-8866-f7cd8b841f99/dns/0.log" Oct 30 22:24:08.466960 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:08.466931 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dspqr_ddc7b666-f4e0-4a1d-8866-f7cd8b841f99/kube-rbac-proxy/0.log" Oct 30 22:24:09.719459 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:09.719430 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hqlzb_59bc969b-f10f-4b5b-a52c-788fda6582e4/dns-node-resolver/0.log" Oct 30 22:24:14.276806 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:14.276772 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq4pk_b75d3c73-4085-468b-ac88-0956a8351130/node-ca/0.log" Oct 30 22:24:19.104368 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:19.104336 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-549f9d654fqblwr_35e7e234-f650-479c-83b9-26e9e9195867/istio-proxy/0.log" Oct 30 22:24:23.031905 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:23.031870 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-6bfzs_d6df4698-b79b-4ef9-9f0f-4a59754b091d/istio-proxy/0.log" Oct 30 22:24:25.131410 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:25.131368 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jq6qc_f76cebce-aa54-4518-ab9d-5e5530a1c3c2/istio-proxy/0.log" Oct 30 22:24:25.547700 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:25.547669 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-qm6kg_f5bcd4b4-0868-47eb-bcc7-3fe245974842/istio-proxy/0.log" Oct 30 22:24:27.227090 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:27.227063 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-95cb67c75-pxxjd_3aeeb498-3174-46e9-a86c-ade577195b67/istio-proxy/0.log" Oct 30 22:24:28.484868 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:28.484835 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6dc5bbb4f7-hmkgf_bc208c6c-8b3d-4eb0-b7e6-ab2daef5578a/router/0.log" Oct 30 22:24:32.435766 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.435727 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 22:24:32.436221 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.435962 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerName="istio-proxy" containerID="cri-o://3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974" gracePeriod=30 Oct 30 22:24:32.484459 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.484418 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 22:24:32.484717 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.484671 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerName="istio-proxy" containerID="cri-o://ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8" gracePeriod=30 Oct 30 22:24:32.486987 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.486956 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 22:24:32.487147 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.487126 4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" containerID="cri-o://ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7" gracePeriod=30 Oct 30 22:24:32.661073 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:32.661043 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qmnhw_7aafa0f5-c7f7-40fc-a6b9-6b4d551bcca2/serve-healthcheck-canary/0.log" Oct 30 22:24:34.852237 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:34.852201 4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jq6qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:34.852646 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:34.852279 4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:35.104587 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:35.104520 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rs27v_5f0ca892-6436-4c1b-994c-bb97ba977307/insights-operator/1.log" Oct 30 22:24:35.162714 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:35.162682 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-rs27v_5f0ca892-6436-4c1b-994c-bb97ba977307/insights-operator/0.log" Oct 30 22:24:35.560853 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:35.560827 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-76zdg_85c676ae-424f-431f-b349-27eeb74c1c1a/kube-rbac-proxy/0.log" Oct 30 22:24:35.973895 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:35.973859 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-76zdg_85c676ae-424f-431f-b349-27eeb74c1c1a/exporter/0.log" Oct 30 22:24:36.387912 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:36.387831 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-76zdg_85c676ae-424f-431f-b349-27eeb74c1c1a/extractor/0.log" Oct 30 22:24:37.541121 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:24:37.541086 4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-76zdg" containerName="extractor" Oct 30 22:24:37.766804 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.766777 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 22:24:37.794719 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.794699 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 22:24:37.797938 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.797918 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 22:24:37.820290 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.820264 4921 generic.go:356] "Generic (PLEG): container finished" podID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" containerID="ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8" exitCode=0 Oct 30 22:24:37.820424 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.820309 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" Oct 30 22:24:37.820424 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.820350 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" event={"ID":"d6df4698-b79b-4ef9-9f0f-4a59754b091d","Type":"ContainerDied","Data":"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8"} Oct 30 22:24:37.820529 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.820441 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs" event={"ID":"d6df4698-b79b-4ef9-9f0f-4a59754b091d","Type":"ContainerDied","Data":"becebced3ee1b978b5f53ec50930b0d756a9cf8874709b1b723c6bc8363edd1d"} Oct 30 22:24:37.820529 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.820476 4921 scope.go:122] "RemoveContainer" containerID="ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8" Oct 30 22:24:37.821775 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.821751 4921 generic.go:356] "Generic (PLEG): container finished" podID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" containerID="ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7" exitCode=0 Oct 30 22:24:37.821872 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.821778 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" event={"ID":"f76cebce-aa54-4518-ab9d-5e5530a1c3c2","Type":"ContainerDied","Data":"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7"} Oct 30 22:24:37.821872 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.821799 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" Oct 30 22:24:37.821872 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.821825 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc" event={"ID":"f76cebce-aa54-4518-ab9d-5e5530a1c3c2","Type":"ContainerDied","Data":"44e571fae25d57bdf6dbca1f38b31c9575cb2b8db412429be119b9ae2c165e46"} Oct 30 22:24:37.823368 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.823349 4921 generic.go:356] "Generic (PLEG): container finished" podID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" containerID="3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974" exitCode=0 Oct 30 22:24:37.823443 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.823381 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" event={"ID":"f5bcd4b4-0868-47eb-bcc7-3fe245974842","Type":"ContainerDied","Data":"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974"} Oct 30 22:24:37.823443 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.823407 4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" Oct 30 22:24:37.823443 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.823426 4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg" event={"ID":"f5bcd4b4-0868-47eb-bcc7-3fe245974842","Type":"ContainerDied","Data":"bd7d08d1470994364b64d025bf7935f790542bba5bfc2a58136e8e6946acff36"} Oct 30 22:24:37.831028 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.831013 4921 scope.go:122] "RemoveContainer" containerID="ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8" Oct 30 22:24:37.831291 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:24:37.831266 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8\": container with ID starting with ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8 not found: ID does not exist" containerID="ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8" Oct 30 22:24:37.831396 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.831303 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8"} err="failed to get container status \"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8\": rpc error: code = NotFound desc = could not find container \"ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8\": container with ID starting with ab702cce5c31c298a352a37a12e1d3a8b406bce98bdc5d9b787ecb4ad51509d8 not found: ID does not exist" Oct 30 22:24:37.831396 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.831354 4921 scope.go:122] "RemoveContainer" containerID="ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7" Oct 30 22:24:37.840124 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.840095 4921 scope.go:122] "RemoveContainer" containerID="ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7" Oct 30 22:24:37.840394 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:24:37.840374 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7\": container with ID starting with ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7 not found: ID does not exist" containerID="ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7" Oct 30 22:24:37.840462 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.840400 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7"} err="failed to get container status \"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7\": rpc error: code = NotFound desc = could not find container \"ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7\": container with ID starting with ca2dadb41480f0bc60368a7f8cf27889ec5922b4686682a79ff14d445da7eca7 not found: ID does not exist" Oct 30 22:24:37.840462 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.840415 4921 scope.go:122] "RemoveContainer" containerID="3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974" Oct 30 22:24:37.848853 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.848832 4921 scope.go:122] "RemoveContainer" containerID="3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974" Oct 30 22:24:37.849091 ip-10-0-137-220 kubenswrapper[4921]: E1030 22:24:37.849073 4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974\": container with ID starting with 3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974 not found: ID does not exist" containerID="3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974" Oct 30 22:24:37.849174 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.849096 4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974"} err="failed to get container status \"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974\": rpc error: code = NotFound desc = could not find container \"3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974\": container with ID starting with 3968c982e7082427777bc53b702b89d2ce03b340f5d7e961d34e9617bc0ba974 not found: ID does not exist" Oct 30 22:24:37.855401 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855378 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.855507 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855406 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.855507 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855423 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.855507 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855443 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.855507 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855470 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.855507 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855497 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855525 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855554 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855601 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855648 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855667 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855679 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855691 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855732 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.855794 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855771 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855839 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855886 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855891 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855950 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855981 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.855987 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856010 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856042 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856072 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856101 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856173 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856210 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\") pod \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\" (UID: \"f76cebce-aa54-4518-ab9d-5e5530a1c3c2\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856243 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.856273 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856271 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856300 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856363 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856373 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\") pod \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\" (UID: \"f5bcd4b4-0868-47eb-bcc7-3fe245974842\") " Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856407 4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\") pod \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\" (UID: \"d6df4698-b79b-4ef9-9f0f-4a59754b091d\") " Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856839 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856862 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856877 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856890 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856893 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856904 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856963 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.857049 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.856986 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.857164 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.857179 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.857551 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.857581 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.857858 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.858046 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.858000 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.859490 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.858055 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.859574 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.859543 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:37.860774 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860749 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.860878 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860777 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "kube-api-access-cm4mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.860878 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860813 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:37.860878 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860850 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.861056 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860882 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.861056 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860904 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.861056 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.860957 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy" pod "d6df4698-b79b-4ef9-9f0f-4a59754b091d" (UID: "d6df4698-b79b-4ef9-9f0f-4a59754b091d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.861259 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.861237 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "kube-api-access-jg8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.861307 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.861270 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx" pod "f76cebce-aa54-4518-ab9d-5e5530a1c3c2" (UID: "f76cebce-aa54-4518-ab9d-5e5530a1c3c2"). InnerVolumeSpecName "kube-api-access-hhqqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:37.861651 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.861635 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:37.862002 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.861985 4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy" pod "f5bcd4b4-0868-47eb-bcc7-3fe245974842" (UID: "f5bcd4b4-0868-47eb-bcc7-3fe245974842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:37.958470 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958442 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958474 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-kube-api-access-hhqqx\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958493 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4mp\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-kube-api-access-cm4mp\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958508 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958521 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958534 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958548 4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-credential-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958573 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958562 4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8rt\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-kube-api-access-jg8rt\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958575 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958588 4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-workload-socket\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958602 4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-data\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958615 4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istiod-ca-cert\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958628 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958641 4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-workload-certs\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958653 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958666 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958679 4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-podinfo\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958692 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76cebce-aa54-4518-ab9d-5e5530a1c3c2-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958705 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bcd4b4-0868-47eb-bcc7-3fe245974842-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958718 4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-envoy\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:37.958786 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:37.958731 4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6df4698-b79b-4ef9-9f0f-4a59754b091d-istio-token\") on node \"ip-10-0-137-220.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:38.326742 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.326702 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 22:24:38.331192 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.331167 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jq6qc"] Oct 30 22:24:38.438353 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.438265 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 22:24:38.445574 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.445543 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-qm6kg"] Oct 30 22:24:38.553576 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.553549 4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 22:24:38.558349 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:38.558328 4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6bfzs"] Oct 30 22:24:39.366967 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:39.366937 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-ldlfp-ztts9_d7078a48-6135-462f-848c-808eb51810d6/insights-gathering/0.log" Oct 30 22:24:40.166444 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:40.166412 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6df4698-b79b-4ef9-9f0f-4a59754b091d" path="/var/lib/kubelet/pods/d6df4698-b79b-4ef9-9f0f-4a59754b091d/volumes" Oct 30 22:24:40.166852 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:40.166839 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bcd4b4-0868-47eb-bcc7-3fe245974842" path="/var/lib/kubelet/pods/f5bcd4b4-0868-47eb-bcc7-3fe245974842/volumes" Oct 30 22:24:40.167239 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:40.167227 4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76cebce-aa54-4518-ab9d-5e5530a1c3c2" path="/var/lib/kubelet/pods/f76cebce-aa54-4518-ab9d-5e5530a1c3c2/volumes" Oct 30 22:24:44.063979 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:44.063942 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6c776cd74c-bk6kw_c7bbe06a-4326-45cd-949d-a0e511b852eb/manager/0.log" Oct 30 22:24:44.477222 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:44.477198 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6c776cd74c-bk6kw_c7bbe06a-4326-45cd-949d-a0e511b852eb/copy-manifests/0.log" Oct 30 22:24:52.593528 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:52.593495 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-nl2n2_dcf552bb-d943-4039-9f25-fd35df68f826/manager/0.log" Oct 30 22:24:53.056803 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:53.056773 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fw5z9_4ba4bda3-21eb-4b27-9c77-2cc7c05707c9/manager/0.log" Oct 30 22:24:54.413113 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:54.413081 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6bf6cbd4cd-tbp75_e26644f3-d7bb-4c37-8c13-a4aacb616d72/manager/0.log" Oct 30 22:24:54.768013 ip-10-0-137-220 kubenswrapper[4921]: I1030 22:24:54.767987 4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vlp2m_74f023e8-abb9-482b-9597-f4e9e60831d8/server/0.log"